10 Gbe startup tips

Hi all. My problem is speed over the network, I started building a truenas box and started to copy a few 10s of gig to it and it took about a week.

I use a 2018 mac mini with 1 gb NIC and the motherboard in my truenas is also 1 gbe.

In the jungle of network solutions & products it`s difficult to know what product to choose so I can get faster transfer.

On my mac I have Thunderbolt & I would like a some product to plug in there that provides a 10 gbe nic.
Without breaking the bank.

Any suggestions?

For the Mac Mini, CalDigit makes some good TB3 products, hereā€™s a TB3 to 10g adaptor: https://www.amazon.com/CalDigit-Connect-10G-Thunderbolt-Ethernet/dp/B07RH7VPDF

Switches there are some others but this NetGear one looks easy enough with 2x 10g ports: https://www.amazon.com/NETGEAR-2x10-Gig-Multi-Gig-Lifetime-Protection/dp/B0765ZPY18
Also found this if you want more 10g ports: https://www.amazon.com/TRENDnet-EdgeSmart-10GBASE-T-Protection-TEG-7080ES/dp/B07BV2VYVF

For the FreeNAS / TrueNAS box, they really like Chelsio or Intel cards, hereā€™s a Chelsio 10g card on Amazon: https://www.amazon.com/Chelsio-T520-BT-Dual-10GbE-Adapter/dp/B07BCNSPFR

Looks like itā€™s around $550 all in for the above

Hi
Thank you for all your suggestions. I really appreciate all the help.

IĀ“m a bit confused about all the different things like the dongle thatĀ“s inserted into the sfp? switch. Do I need dongles to insert into the network card to use common network cable?

IĀ“ll look into the links.

Depending on the length of cable needed, there are copper cables with an SFP type plug integrated into each end those tend to be no longer than 3 meters I (think?). These are sometimes referred to as Direct Attached SFP copper.

SFP are simply media converters allowing either a copper SFP port or a fiber port (for much longer runs), the electronics on these determine what speeds it will handle. SFP ā€˜donglesā€™ are only relative to SFP enabled switch gear.
If you buy SFP enabled switch gear make sure you buy a supported SFP type, some switches are specific about what they will or wonā€™t support.

None of this relates directly to a thunderbolt product, unless you are going to use some type of thunderbolt to ethernet adaptor. Apple is notorious for not letting itā€™s devices interact with anything it deems incompatible or unsupported (same thing).

Cal Digitā€™s thunderbolt adapter is not SFP enabled. If you are not going to use any media except for copper just skip SFP gear entirely.

This NIC would require SFP ā€˜donglesā€™ https://tinyurl.com/yy89hflh, itā€™s an intel 2 port 10 gigE.

I prefer 4 ports, channel bonding at least 2 of those ports into one virtual ethernet cable is a nice option to have.

Startech makes a 2 port for 280-ish.

I used to source what they call ā€˜working pullsā€™ from ebay, basically network cards pulled in working condition from machines being overhauled. They were sold at super cheap prices typically.

Hi
IĀ“ll stick to the easiest solution there is. Just a network cable without any dongles or special connections. Will this NIC do the job as I intent it for in a truenas solution? Directly connected into my mac thru a Thunderbolt to 10 Gbe link?

Is it supported?
https://www.amazon.co.uk/10GtekĀ®-10GbE-Network-Intel-X540-T2/dp/B077Z318JF/ref=psdc_949408031_t3_B07BCNSPFR?th=1

I appreciate your tips & your time very much. It helps he alot to get backup from my mac mini.

Thank you

In this case I did everything with RJ45 ethernet as itā€™s easier to work and likely something youā€™re familiar with already, vs SFP+ optics or DACs which are normally used with switches and servers as opposed to consumer gear.

Youā€™d connect the TB3 dongle to your Mac, this will act as a network adaptor. Install the 10g card into your TrueNAS box. From there, plug a CAT6/6A cable from the TB3 dongle into the 10g port on the switch, as well as another CAT6/6A from the TrueNAS box to the 2nd 10g port on the switch. Connect the switch to your main network, either via another switch or your router, depending on your setup.

Both of these connections would be your main connections for both devices to your network.

Iā€™m not familiar with that specific model, but it looks like it uses an Intel X550 which would be supported fine with TrueNAS.

You can directly connect your Mac (via the TB3 adaptor) to the TrueNAS box and skip the switch altogether, but youā€™d need to setup static IP addresses in a different subnet than your normal network.

IE: if your network is 192.168.2.0-254 youā€™d want to put the IPs for the 10g links in say 192.168.100.0-254, this way the traffic between them will go over the 10g links instead of the 1g links. This will only work if you use the IPs on the 10g links, if you connect to the IP of the 1g link, youā€™ll get 1g speeds.

This issue is why I recommended a switch as it simplifies the configuration slightly since you no longer need to utilize static IPs for the 10g links.

Your problem may not be network speed, I commonly move terabytes and never need to wait more than 4 -5 hours, even if it is a lot of small files. Small files take more time than large things like contiguous video files (sort of). And all I have is gigabit to the clients. Iā€™d take a look at what your throughput is and see if it is close to wire speed. You should be able to get 900mbps (100+MBps) unless the bottleneck is somewhere else, or you are moving thousands of small text files and pictures. When moving thousands of small files, I may only get a quarter of the speed depending on the files. A 10gbe connection between server and client may not help much with things like this.

Somehow I glossed over the ā€œ10s of gigsā€, must have read TBs or something else.

  • What is the config of the TrueNAS box? CPUs, disks, RAID configuration, etc.
  • Run an iPERF test between the Mac Mini and the TrueNAS box

For TrueNAS, either SSH into the server or in the GUI go to Shell, run iperf -s to start the server. On the Mac, download iPERF 2, Iā€™m not 100% on Mac instructions but the command is ipefc -c [server IP]

On the server you should see the network speed the 2 can communicate at, below is what mine looks like. The first result is what you should see running at 1gbe, the 2nd is 10gbe.

image

Hi Greg. I tried to move an photo library wich is about 900 Gb over and I think the NIC on the motherboard reset itself a few times or the file is just to big.
I`ve ordered an thunderbolt 3 to 10 Gbe adapter and a 10 Gb nic for my truenas. Hopefully I can take a full backup

Thank you for your reply & suggestions
I really apriciate all input on this topic

Hi.

I feel this flew right over my technical level. Ill have to come back to you in a couple of days on this. Ill try to find it out

Thank you for your tips, suggestions & time

OK, so thousands of ā€œsmallā€ files, yes that might take a long time, and it is very likely that 10gbe wonā€™t fix anything. Really need to know if you are transferring at maximum speeds or not. Copy some nice big video files over, stuff over 4 gigabytes and see how long that takes, even better if you can get a real speed metric like from the FreeNAS pages to see how much bandwidth is being used during these transfers.

Now if there is a hardware fault too, then that is something that needs to be addressed.

Also what is you disk configuration, data transfer is a two sided thing, if you have a slow drive on either side it doesnā€™t matter how fast the network can move the data. We get that when we move video files from our cameras to our storage, the SD cards only go as fast as they go.

Yet if I move those same files from server back to local hard drive they move pretty good (lots of database files go with the clips). If I take a large export from a video project, moving this is fast, same with zipped files and I normally ā€œsaturateā€ he gigabit connection going to each client. I was able to do that when the server was on gigabit, 2 gigabit (aggregated), and now 2 10gbe (aggregated). While 10gbe to some clients would be nice, it just isnā€™t important to what most people are doing for a lot of different reasons. The number of people working with uncompressed video is relatively small and certainly needs fast storage too.

1 Like