So, I never could get a connection the way the cards came. So I tried two things. First, in the cards BIOS, there seems to be 2 major mode settings. One I think was called SH and it seems to be a single connection with all of the bandwidth. The second was called partitioned mode. When I set it to this, I now have 4 connections per interface, each with two MAC addresses, a primary with the full card expected MAC and the second is the same MAC with the first two bytes set to 00 00. Suddenly, some of the interfaces sort of worked - well enough to pull an IP from the PFsense box. They still show as no carrier and Network Manager will not allow them to become active. So I ordered some DAC cables and SFP+ from a different manufacturer, thinking there could be some issue with either the fiber or the SFP+ modules. Swapping the SFP+ made no difference. The DAC cables did make a difference! Still have no carrier, but now the interface came active in Network Manager. Despite the interface being active and having pulled an IP from the DHCP server on the PFsense box, I still cannot ping either the gateway address or another interface address. Guessing this has to do with the interface not getting the full data, gateway address, route information, etc.
The card in all 3 servers are HEWLETT PACKARD 652503-B21 530SFP+ dual port.
Should this be set in the Partitioned mode? Or the other one?
I suspect the other mode is designed for iSCSI as I see iSCSI references in the log. iSCSI itself is not set up or running as best I know.
Comments on card mode and Network Manager set up would be appreciated.