I’m trying to upgrade part of my home network to 10gb – mainly my TrueNAS PC and the Ubuntu 24.04LTS PC that will be processing data on the NAS – and I can’t get a link established at 10gb, and 1GB only occasionally. Here’s my setup:
TP-Link TL-SG3428X. This has 4 SFP+ ports, which docs say work at 1GB or 10GB. It also advises you shouldn’t put more than one RJ45 transceiver in the SFP+ slots, which is why I decided to use fiber.
2 TP-Link SM5110-SR transceivers, one to connect to the TrueNAS PC, the other to connect to the Ubuntu PC. I read that you should match the transceivers to the controller, so TP-Link transceivers for the SG3428X .
2 10gtek X520-10G-1S-X8 pcie cards, one going in the Ubuntu PC, the other going in the TrueNAS PC. These say they are Intel 82599EN controllers, equivalent to X520-DA1 cards.
2 10gtek AXS85-192-M3 SFP+ SR transceivers, “Intel coded”. Again, got Intel transceivers to match Intel controllers. These go into the 10gtek cards.
2 4m LC to LC Fiber Cable, OM4 4M Fiber Optic Cable Multimode Duplex 50/125um LSZH 40GB/100GB, one to connect the Ubuntu PC to the switch, the other for the TrueNAS PC.
Started by just connecting the Ubuntu PC, to test. Put a TPlink transceiver in port 28 of the SG3428X, and set it up for 10gb full duplex, flow control enabled. Added a 10gtek pcie card to the Ubuntu PC, inserted a 10gtek tranceiver into that, and connected fiber cable between the two. The 10gtek card was recognized as Intel 82599 chipset, loaded the ixgbe driver, and syslog reported the driver started up as 10gb, but there was no link led on the switch or NIC.
Started trying to diagnose the problem. With the switch and NIC set to 10gb, full duplex, flow control enabled, I could not get a link. Ubuntu side would up the interface briefly, and shut it down with no carrier. At one point, I was able to get a link by setting the switch port to 1GB and the NIC to 1GB, and successfully activated the interface. But I can’t repeat that consistently. Even without a link, or carrier, I think the switch and NIC are seeing each other in some way, because if I place the NIC on auto-negotiate, I could change the switch port between 1gb and 10gb and the Ubuntu would change to match. But I am stuck getting a reliable link to being the NIC interface up.
Any advice on debugging this problem? I still have time to return the 10gtek parts if there is a better, or more correct, option. I could also switch to RJ45 10gb between the switch and one of the PCs, but TP-Link cautions against more than one of those in the switch, so I still need the rest to be fiber. I could also try putting the other NIC in the TrueNAS PC, and test if that will work to the switch or even between the TrueNas and Ubuntu PCs. Theoretically I could have a bad part, but I also read these things can be “persnickety”.