Xen Orchestra SMB vs NFS remote speeds (TrueNAS)

Computer Hardware & Server Infrastructure Builds
1

Hey folks,

I’ve been doing some testing, and it seems for whatever reason, that SMB is faster?

I thought NFS was better in linux to linux communication. Perhaps SMB is doing some sort of caching since the expected maximum speed for the Seagate Barracuda drives is about 150 MB/s, so I doubt it’s the drive speed that’s being saturated on every test run.

Was just curious what everyone else uses?

Results:

XOA writing to NFS share mount directory:

for i in {0..5}; do dd if=/dev/urandom of=testfile.bin bs=1M count=1000; done
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.36581 s, 125 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 10.0539 s, 104 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 9.31783 s, 113 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.67909 s, 121 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.96784 s, 117 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.65976 s, 121 MB/s

XOA writing to SMB share mount directory:

for i in {0..5}; do dd if=/dev/urandom of=testfile.bin bs=1M count=1000; done
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 5.93853 s, 177 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.08925 s, 172 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.16683 s, 170 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.20099 s, 169 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.01749 s, 174 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.1518 s, 170 MB/s

Both devices (TrueNAS Scale VM, and a Debian Xen Orchestra VM) are connected over Netgate 4100 (2.5G) - And are technically running as VMs on the same hypervisor, but on different VLANs, so all communication has to go over pfsense.

Iperf3 results (TrueNAS as client, XOA as server):

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec  3.09 GBytes   884 Mbits/sec   44             sender
[  5]   0.00-30.06  sec  3.09 GBytes   882 Mbits/sec                  receiver
[  7]   0.00-30.00  sec  1.61 GBytes   462 Mbits/sec   56             sender
[  7]   0.00-30.06  sec  1.61 GBytes   461 Mbits/sec                  receiver
[  9]   0.00-30.00  sec  1.72 GBytes   491 Mbits/sec   15             sender
[  9]   0.00-30.06  sec  1.71 GBytes   490 Mbits/sec                  receiver
[ 11]   0.00-30.00  sec  1.74 GBytes   500 Mbits/sec   18             sender
[ 11]   0.00-30.06  sec  1.74 GBytes   498 Mbits/sec                  receiver
[SUM]   0.00-30.00  sec  8.16 GBytes  2.34 Gbits/sec  133             sender
[SUM]   0.00-30.06  sec  8.15 GBytes  2.33 Gbits/sec                  receiver

TrueNAS - Seagate Barracuda 3.5" 2TB - ST2000DM008; Raid 1

for i in {0..5}; do dd if=/dev/urandom of=testfile.bin bs=1M count=1000; done
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 2.65334 s, 395 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 5.16069 s, 203 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 6.38631 s, 164 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.13018 s, 129 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 7.27549 s, 144 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.44536 s, 124 MB/s

dd if=/dev/urandom of=testfile.bin bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB, 4.9 GiB) copied, 33.8633 s, 155 MB/s

Anyways, the goal of my research: I use rsync to synchronize everything from those 2 raid drives to a Raspberry Pi with a 4TB WD Red Plus, and the ZFS lz4 compressed share with XOA backups on TrueNAS, has been having oddly slow transfer speeds to the RPi, just 25 MB/s.

Synchronizing all other shares seems normal, although I’m not sure if it’s the compression or something else, since that share is the one that gets the most writes to it daily, and is the biggest one out of all (1TB).

It’s definitely not the write speeds on the RPi (which should be about 175MB/s).
It’s definitely not the network link between TrueNAS and the RPi.
There doesn’t seem to be a lack of processing power, since neither TrueNAS, nor the RPi go above 30% CPU usage during the transfer.
So what the hell kind of trickery could it be?

RPi write speeds to the WD Red Plus 4TB (note how I used /dev/zero - RPi seems to be choking with /dev/urandom and giving me only about 45MB/s - Maybe this is a part of the bigger problem?):

for i in {0..5}; do dd if=/dev/zero of=testfile.bin bs=1M count=1000; done
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 5.94795 s, 176 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 7.87396 s, 133 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 7.91027 s, 133 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.24003 s, 127 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.2287 s, 127 MB/s
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 8.21602 s, 128 MB/s

Iperf3 server results (TrueNAS as client, RPi as server): This is all good since the RPi only has a 1Gbit link

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-30.00  sec   616 MBytes   172 Mbits/sec   66             sender
[  5]   0.00-30.09  sec   613 MBytes   171 Mbits/sec                  receiver
[  7]   0.00-30.00  sec   589 MBytes   165 Mbits/sec   62             sender
[  7]   0.00-30.09  sec   587 MBytes   164 Mbits/sec                  receiver
[  9]   0.00-30.00  sec   486 MBytes   136 Mbits/sec   26             sender
[  9]   0.00-30.09  sec   484 MBytes   135 Mbits/sec                  receiver
[ 11]   0.00-30.00  sec  1.64 GBytes   470 Mbits/sec    0             sender
[ 11]   0.00-30.09  sec  1.64 GBytes   469 Mbits/sec                  receiver
[SUM]   0.00-30.00  sec  3.29 GBytes   943 Mbits/sec  154             sender
[SUM]   0.00-30.09  sec  3.29 GBytes   938 Mbits/sec                  receiver
2

Routing storage is never a great option because you create a slow down at the pfsense/routing part for moving the storage.

3

How do you suggest I get around that?

Obviously I’d need a physical storage server rather than a VM, but shouldn’t Storage sit in it’s own VLAN and subnet, for security reasons?

Perhaps adding a switch under the netgate box?

4

Our XO instance has two network interfaces, one for talking to XCP-NG and the other for talking to storage.

5

MSP’s HATE HIM! This simple trick will double your backup speeds! (PS. Thank you!)

image

6

MSP’s hated who? Also please elaborate what trick are you pertaining to?

7

It’s a joke. You’ve surely heard those clickbait-y commercials: “X hates Y, this simple trick will solve all your problems!”

By “Him” I mean Tom and by “Trick” I mean him telling me to put two interfaces on my XO, which seems to have doubled the backup speeds.

8

Ah, sorry not really familiar those type of jokes.

9

Tom, this would be great to include in a video sometime. I’m building out some data systems and I’d like to see a clear explanation of best practices for talking to storage on XOA.

10

I have a storage video

Backups