I have a bit of a strange problem with transferring data between my two FreeNAS boxes.
One of them is my primary box, it has an LSI 9211-8i in IT mode with 8x4TB drives in RAIDZ2. The machine has 16GB RAM and the CPU is a quad core Xeon. It has a dual port Intel X520 network card for 10Gbe connections.
The other is my backup box, it has an LSI 9211-8i in IT mode with 6x3TB drives in RAIDZ1. The machine has 16GB RAM and the CPU is a quad core Xeon. It has a dual port Intel X520 network card for 10Gbe connections.
If I use my desktop to connect directly to either of the NAS boxes I get good transfer speeds over SMB (approx 600MB/s) so I know the direct connections, the cables, and the storage speed of both boxes is more then good enough to get somewhere near the 10Gbe limit. The desktop has a single port variant of the same network card as the NAS boxes.
Running iPerf on the FreeNAS boxes I get 9.8ish Gb/s in both directions, so all working as expected.
However, if I use rSync or do a direct file copy using cp or pv, I seem to be limited to approx 125MB/s. My main reason for going 10Gbe was for backing up my main NAS to my backup NAS, so I’m a little dissappointed. I use the rSync daemon usually but I’ve also tried passing rSync over SSH (this was actually a little slower). cp and pv I did over NFS mounts.
For reference, the rSync commands look like;
/usr/local/bin/rsync -artW --delete --progress --omit-dir-times --no-compress --inplace /mnt/data_volume/Media/TV\ Shows/ 192.168.10.131::Media/TVShows/
I’ve recently added --no-compress and --inplace for testing, but they make very little difference and I will probably remove them soon.
I haven’t tested ZFS replication. The backup isn’t one to one as I back up more important stuff to the cloud.
Anyone got any insight as to why speed between the two FreeNAS boxes is so much slower than expected?