10gb fiber home setup issues

Networking & Firewalls
1

I’m trying to upgrade part of my home network to 10gb – mainly my TrueNAS PC and the Ubuntu 24.04LTS PC that will be processing data on the NAS – and I can’t get a link established at 10gb, and 1GB only occasionally. Here’s my setup:

TP-Link TL-SG3428X. This has 4 SFP+ ports, which docs say work at 1GB or 10GB. It also advises you shouldn’t put more than one RJ45 transceiver in the SFP+ slots, which is why I decided to use fiber.

2 TP-Link SM5110-SR transceivers, one to connect to the TrueNAS PC, the other to connect to the Ubuntu PC. I read that you should match the transceivers to the controller, so TP-Link transceivers for the SG3428X .

2 10gtek X520-10G-1S-X8 pcie cards, one going in the Ubuntu PC, the other going in the TrueNAS PC. These say they are Intel 82599EN controllers, equivalent to X520-DA1 cards.

2 10gtek AXS85-192-M3 SFP+ SR transceivers, “Intel coded”. Again, got Intel transceivers to match Intel controllers. These go into the 10gtek cards.

2 4m LC to LC Fiber Cable, OM4 4M Fiber Optic Cable Multimode Duplex 50/125um LSZH 40GB/100GB, one to connect the Ubuntu PC to the switch, the other for the TrueNAS PC.

Started by just connecting the Ubuntu PC, to test. Put a TPlink transceiver in port 28 of the SG3428X, and set it up for 10gb full duplex, flow control enabled. Added a 10gtek pcie card to the Ubuntu PC, inserted a 10gtek tranceiver into that, and connected fiber cable between the two. The 10gtek card was recognized as Intel 82599 chipset, loaded the ixgbe driver, and syslog reported the driver started up as 10gb, but there was no link led on the switch or NIC.

Started trying to diagnose the problem. With the switch and NIC set to 10gb, full duplex, flow control enabled, I could not get a link. Ubuntu side would up the interface briefly, and shut it down with no carrier. At one point, I was able to get a link by setting the switch port to 1GB and the NIC to 1GB, and successfully activated the interface. But I can’t repeat that consistently. Even without a link, or carrier, I think the switch and NIC are seeing each other in some way, because if I place the NIC on auto-negotiate, I could change the switch port between 1gb and 10gb and the Ubuntu would change to match. But I am stuck getting a reliable link to being the NIC interface up.

Any advice on debugging this problem? I still have time to return the 10gtek parts if there is a better, or more correct, option. I could also switch to RJ45 10gb between the switch and one of the PCs, but TP-Link cautions against more than one of those in the switch, so I still need the rest to be fiber. I could also try putting the other NIC in the TrueNAS PC, and test if that will work to the switch or even between the TrueNas and Ubuntu PCs. Theoretically I could have a bad part, but I also read these things can be “persnickety”.

2

Have you tried to bring up a link with the same model transceiver on both ends?

3

Yes; I put the other TP-Link transceiver into the 10gtek card on the Ubuntu PC, and got nowhere. No link at either speed. I assume that’s because of what I read about needing to keep transceiver brand matching device brand.
A small update: I think I now have a stable connection, but only at 1gb, between the switch and the Ubuntu PC. Not quite sure what I did beside clear things out and try again. But, any attempt to change things to 10gb refuses to link.

4

I don’t have much experience with TP Link switches but they should work fine at 10G. Fiber is sensitive to any dust getting on the optic and can certainly cause link issues. DAC is probably an easier choice for a setup like this.

1 Like
5

You could have a bad SFP+ module. I have that same 10Gtek card in one of my Truenas Core machines, no issues with it. No issues with different types of modules, I did have one fail after installing that card but it’s been good since (about 2 years now).

How much distance is really between the switch and the device? If withing a couple meters, I’d try a DAC cable (another 10Gtek product), I have several in use on my production system and my lab.

I’m also a big fan of buying used “Cisco” branded SFP+ modules on ebay, normally around $8usd and I think I’ve only had one that went bad. I have a bunch of these for my lab and they’ve been solid in different cards and switches.

1 Like
6

The TrueNAS box could be reached with about 4 feet of cable, the Ubuntu PC with about 5 feet.
I’m willing to try DAC, will look for “pre-made” (DAC with attached transceivers).

On the issue of “vendor lock”, which is why I made sure I got TP-Link transceivers for the switch, and 10gtek “Intel Coded” for the 10gtek cards; am I understanding that whole issue correctly? That is, TPlink really wants a TPlink transceiver in it, and the 10gtek (Intel 82599) really wants an Intel transceiver in it? Seems that might make it hard to find a DAC cable to go between the two.

I’m also going to put my other 10gtek NIC into the TrueNAS PC, and see if it works better, and I can also then connect the TrueNAS PC directly to the Ubuntu PC and see if the 10gtek cards see each other better.

7

You are correct about vendor lock, but I would hope TPLink doesn’t do that. I have not discovered a vendor lock with the 10Gtek card in my server, so I think you are OK there. I just checked and I’m using 10Gtek branded DAC cables, and I’ve also used several different multimode fiber modules with this card. Mikrotik switches have no issues with these DAC cables.

If not, I think you can maybe use the ethtool command to reprogram your card, or get all transceivers approved firmware for the card. Serve the Home has a couple of very long threads about fixing Intel NIC card issues like this.

Connecting the truenas directly to the Ubuntu is another good idea to check for functions, as long as the IP address is the same network, you should at least be able to talk to each other by IP address.

1 Like
8

I have both Chelsio T520 and Intel 520 both using DAC and neither card seem to care what DAC model I use. Also my UniFi and MikroTik switches do not seem to care what DAC transceiver I use either.

1 Like
9

Good to know. I’ll get a couple of three meter DAC cables ordered. I’ll check “Serve the Home” to learn the ethtool reprogramming bits. And will see if direct-connecting the two PC’s together reveals anything.
I hope it’s not the TP-Link switch, it’s served me well, and works well with the other TP-Link Omada equipment I have (indoor/outdoor access points).
Appreciate the advice; off to diagnose, tinker, and learn!
PS: why am I bothering with 10g? Well, besides that I can, it’s in support of my astrophotography hobby. I store the raw data on the NAS, move it to the PC to process, and store the processed data back on the NAS. Lots of data going around, and want to speed that process up.

10

OK, potentially big amounts of data. Do you capture directly to the NAS or use a camera card to store everything and dump it to the nas when you are done?

What resolution, and are you using Jpeg, Tiff, or RAW on the camera? Just trying to get a sense of the amount of data you are pushing.

11

Don’t record direct to NAS. Record on removable media, transfer to NAS as soon as possible, in keeping with the “Lots Of Copies Keeps Stuff Safe” principle, lol. Resolution varies, up to 24mp. Format is RAW or FITS. I’ll regularly approach 100..200gb raw data in a session. Then, I copy that blob down to the Ubuntu PC for processing. The results get saved back to the NAS. Solar data frequently end up as video animations, which I save the editing work to the NAS if I need to come back to it for changes.
Takes a while to move a couple hundred gig over my existing 1gb lan. NAS is backed up remotely to another NAS (not worried about speeding that up, TrueNAS is pretty efficient at it).

On the whole, I have ended up shoveling almost a terabyte on some occasions.

12

Yeah, 10g would be almost a requirement for that stuff.

[edit] had to look up FITS

13

Well, this is interesting. I put the 2nd 10gtek NIC into the TrueNAS box. Put the other 10gtek transceiver into that, and for a first test connected that to the other TPlink transceiver in the switch. That was just to see if the TrueNAS to switch would happen to work. LOL. TrueNAS box came right up and connected at 10g to the switch. I need to do a little configuring of the interface (DHCP, jumbo frames, etc), but it’s already doing better than Ubuntu-PC to switch. I will likely still try straight from TrueNAS to Ubuntu, but this is starting to lend credence to the “something I got was DOA” theory. Will putter about with the TrueNAS-switch connection, and eventually disconnect the existing 1gb RJ45 and leave it with just the 10gb connection to the switch. Progress! Sort of.

14

FITS, yeah it’s a standard format for astrophotography pictures. Usually used with dedicated astro-cams. RAW format is when I use a DSLR (regular) camera.

15

Another update:
Tried to direct connect the TrueNAS PC to the Ubuntu PC without the switch in the path. Would not link. No lights. The Ubuntu PC could “see”, inasmuch as ethtool on the Ubuntu side showed 10gb, and was seeing laser power, but no link or carrier.

So, looks like either the 10gtek transceiver is bad, and/or the 10gtek NIC is bad. I guess I could swap tranceivers between the Ubuntu PC and the TrueNAS PC, and if the TrueNAS PC lost connectivity, that would point to a bad transceiver. If it worked, points to a bad NIC.

16

'nother update:
Swapped the transceivers between the TrueNAS and Ubuntu PCs. TrueNAS connected as usual. By process of elimination, the card in the Ubuntu PC must be bad. Will be replacing that. 10Gtek DAC cable on the way, too, to give that a try.
All in all, a good learning experience. When it works, it works easily. When it’s broke, diagnose and isolate the broken part.
Thanks for the help!

17

To close this thread out, it ended up being a bad NIC. Replaced that in the Ubuntu PC, used a DAC cable between it and the switch, and it came up instantly. So all is well. Getting 250 to 350 megabytes per second transfer between the TrueNAS PC and Ubuntu PC via SMB. Don’t have jumbo frames turned on yet, my first shot at that produced erratic results, so back to standard MTU for now. Still, faster than it was!

18

Good to hear you have it solved, I would not bother with jumbo frames / MTU > 9000 because with modern hardware the gains are really only 1-3%.

1 Like
19

Alrighty, will – erm, won’t – do, then! Thanks again for the help.