Slow speed between two FreeNAS boxes

I have a bit of a strange problem with transferring data between my two FreeNAS boxes.

One of them is my primary box, it has an LSI 9211-8i in IT mode with 8x4TB drives in RAIDZ2. The machine has 16GB RAM and the CPU is a quad core Xeon. It has a dual port Intel X520 network card for 10Gbe connections.

The other is my backup box, it has an LSI 9211-8i in IT mode with 6x3TB drives in RAIDZ1. The machine has 16GB RAM and the CPU is a quad core Xeon. It has a dual port Intel X520 network card for 10Gbe connections.

If I use my desktop to connect directly to either of the NAS boxes I get good transfer speeds over SMB (approx 600MB/s) so I know the direct connections, the cables, and the storage speed of both boxes is more then good enough to get somewhere near the 10Gbe limit. The desktop has a single port variant of the same network card as the NAS boxes.

Running iPerf on the FreeNAS boxes I get 9.8ish Gb/s in both directions, so all working as expected.

However, if I use rSync or do a direct file copy using cp or pv, I seem to be limited to approx 125MB/s. My main reason for going 10Gbe was for backing up my main NAS to my backup NAS, so I’m a little dissappointed. I use the rSync daemon usually but I’ve also tried passing rSync over SSH (this was actually a little slower). cp and pv I did over NFS mounts.

For reference, the rSync commands look like;

/usr/local/bin/rsync -artW --delete --progress --omit-dir-times --no-compress --inplace /mnt/data_volume/Media/TV\ Shows/

I’ve recently added --no-compress and --inplace for testing, but they make very little difference and I will probably remove them soon.

I haven’t tested ZFS replication. The backup isn’t one to one as I back up more important stuff to the cloud.

Anyone got any insight as to why speed between the two FreeNAS boxes is so much slower than expected?

Have you set the MTU settings on your 10Gb network interfaces to 9000?

You don’t have zfs dedupe on do you?

Yes. I have the mtu set at 9000 in the interface options.

No, zfs dedupe is not on. Like I said both boxes work perfectly fine with my windows desktop. They’re just slow talking to each other using anything other than iperf.

I hardly ever use rsync so I would give ZFS replication a try as it is a more efficient way to move the data.

Have you tried rsync to/from your desktop? Rsync is quite evil when it comes to metadata, so you might be hitting max iops on your systems.

When the rsync is running, take a look at how busy your disks are. Might give an indication of where to look further.

It’s not just rsync though, using cp or even pv also yields the same slow speeds.

I’ve tried using the age old suggested tunables which didn’t improve anything. I’ve removed them as I believe FreeNAS is supposed to be better optmised for 10Gbe traffic now.

Overnight I tried another test. I installed Ubuntu on my desktop and setup an rsync server. I then rsynced a backup set to the Ubuntu machine and got good speeds (approx 500-550 MB/s). I then rsynced the files from the Ubuntu machine to the backup freenas and got similar speeds.

At the minute it’s far quicker for me to put an intermediary machine between the FreeNAS servers to complete a backup.

Something is definitely amiss in the way freenas is communicating with freenas, but I can’t put my finger on it.

I’ve just realised a typo in my original post. All the 10Gb nics are Intel X520 based, not X550! I’ll edit the original post to fix.