Fast iperf, slow transfers. Can't figure it out

Computer Hardware & Server Infrastructure Builds
1

It seems like several other folks have had this problem, just dont see a resolution yet.

I have two servers with Intel X520 dual SFP+ 10G NICs in them. They are just connected directly with a DAC cable. No switch. One server has TrueNAS Dragonfish 24.04 on it. The other has plain vanilla Ubuntu 22.04.4. Both on bare metal, no virtualization. More server specs below.

Iperf3 shows me a solid 9.9 Gbps. But when I tell iperf to actually send a file, things get yucky.

When I iperf a file from Ubuntu to TrueNAS, the transfer rate starts off OK, but calls off fast like this (not to a pool, just to the mirrored boot SSDs):

root@vm-1:~# iperf3  -F ./10gfile.iperf -c 10.8.8.10
Connecting to host 10.8.8.10, port 5201
[  5] local 10.8.8.11 port 37110 connected to 10.8.8.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.04 GBytes  8.95 Gbits/sec   11   1.53 MBytes
[  5]   1.00-2.00   sec  1.08 GBytes  9.26 Gbits/sec    0   1.53 MBytes
[  5]   2.00-3.00   sec   936 MBytes  7.86 Gbits/sec    0   1.54 MBytes
[  5]   3.00-4.00   sec   531 MBytes  4.46 Gbits/sec   19   1.54 MBytes
[  5]   4.00-5.00   sec   341 MBytes  2.86 Gbits/sec    0   1.54 MBytes
[  5]   5.00-6.00   sec   301 MBytes  2.53 Gbits/sec    0   1.54 MBytes
[  5]   6.00-7.00   sec   308 MBytes  2.58 Gbits/sec    0   1.54 MBytes
[  5]   7.00-8.00   sec   334 MBytes  2.80 Gbits/sec    0   1.54 MBytes
[  5]   8.00-9.00   sec   351 MBytes  2.94 Gbits/sec    0   1.54 MBytes
[  5]   9.00-10.00  sec   340 MBytes  2.85 Gbits/sec    0   1.54 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  5.48 GBytes  4.71 Gbits/sec   30             sender
        Sent 5.48 GByte / 10.0 GByte (54%) of ./10gfile.iperf
[  5]   0.00-10.00  sec  5.48 GBytes  4.70 Gbits/sec                  receiver

The other direction is even worse. This is from TrueNAS to Ubuntu (not from a pool, just from its boot SSDs):

root@truenas[~]# iperf3  -F ./10gfile.iperf -c 10.8.8.11
Connecting to host 10.8.8.11, port 5201
[  5] local 10.8.8.10 port 49892 connected to 10.8.8.11 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  11.6 MBytes  97.5 Mbits/sec    2    839 KBytes
[  5]   1.00-2.00   sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]   2.00-3.00   sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]   3.00-4.00   sec  6.25 MBytes  52.4 Mbits/sec    0    839 KBytes
[  5]   4.00-5.00   sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]   5.00-6.00   sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]   6.00-7.00   sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]   7.00-8.00   sec  6.25 MBytes  52.4 Mbits/sec    0    839 KBytes
[  5]   8.00-9.00   sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]   9.00-10.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  10.00-11.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  11.00-12.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  12.00-13.00  sec  6.25 MBytes  52.4 Mbits/sec    0    839 KBytes
[  5]  13.00-14.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  14.00-15.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  15.00-16.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  16.00-17.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  17.00-18.00  sec  6.25 MBytes  52.4 Mbits/sec    0    839 KBytes
[  5]  18.00-19.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  19.00-20.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
[  5]  20.00-21.00  sec  5.00 MBytes  41.9 Mbits/sec    0    839 KBytes
... killed it

rsync over ssh transferred the same 10 GB file at about 180 MB/s which is still pretty lame for a 10G network. NFS performance is similarly bad.

I’ve swapped out the NICs and the DACs. Got no clue. Any idears?

Here’s more server details.

Truenas Server Specs:

  • Dell X720xd
  • 256G ECC RAM
  • 2x E5-2680 V2 2.8GHz 20/40 cores/threads total
  • LSI SAS2308 (Dell H710p Mini Mono in IT mode) running at PCIe 3.0 speeds (x8 I think)
  • 12x HGST 4TB 7.2k SAS, 512 native sectors
  • Intel Optane 900p PCIe card
  • Intel X520 dual port 10G NIC
  • Intel I350 dual port 1G NIC
  • TrueNAS DragonFish 24.04
  • ZFS pool is 2x 6-drive raidz2 vdevs with Optain SLOG.

Ubuntu Server Specs:

  • Dell X720
  • 256G ECC RAM
  • 2x E5-2640 v2 2Ghz 16/32 cores/threads total
  • LSI SAS2308 (Dell H710 Mini Mono in IT mode) running at PCIe 3.0 speeds (x8 I think)
  • 8x HGST 4TB 7.2k SAS, 512 native sectors (not used), boots of a couple of 256G software raided SATA m.2 SSDs
  • Intel X520 dual port 10G NIC
  • Intel I350 dual port 1G NIC
  • Ubuntu Server 22.04.4
2

Since this is the release of 24.04 of TrueNAS you might reach out to them on their forum or see if there is already a bug created for this issue on their Jira.

On thing you can try is disabling async on your zfs pool and see if the performance gets better.

3

Pretty sure your network test with iPerf runs in memory so storage performance issues on your disks wouldn’t surface in your testing. I would say your network looks fine and no need to change NICs or cables.

I think you might be hitting the max limit of IOPS for writes that either your HBA/RAID card or hard drives support. I would check those specs. Also, if you are writing large amounts of data in your testing you will likely fill up the cache built into the drives and this will force writes to have to go straight to the disk which will probably be your lowest ceiling when it comes to performance.

4

Just saw this! Yes you are totally right! A big DUH moment for me. See my next post.

5

So, I think I figured out the iperf issue. Thankfully, it makes total sense.

BLUF: When iperf is just sending data and not transferring files, NO DISKS ARE INVOLVED!!! DUH!

So, the two hosts are now both running truenas. I figured that plain vanilla ubuntu server might not be as finely tuned to this stuff as truenas would be. Dunno. Using truenas on both sides DID seems to make things much better.

One server is named nas-1 and the other vm-1. Both are Dell R720s with 256G RAM. No ZFS pools involved except their mirrored boot drives (SSD on nas-1, HDD on vm-1).

So here is a plain iperf from vm-1 to nas-1:

root@vm-1[~]#  iperf3 -t 20 -c nas-1
Connecting to host nas-1, port 5201
[  5] local 10.8.8.11 port 58576 connected to 10.8.8.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.16 GBytes  9.92 Gbits/sec   44   1.55 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.55 MBytes
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.57 MBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.57 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.91 Gbits/sec    1   1.57 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.59 MBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.60 MBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.66 MBytes
[  5]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.66 MBytes
[  5]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.91 MBytes
[  5]  10.00-11.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.91 MBytes
[  5]  11.00-12.00  sec  1.15 GBytes  9.91 Gbits/sec    0   1.91 MBytes
[  5]  12.00-13.00  sec  1.15 GBytes  9.90 Gbits/sec    1   1.91 MBytes
[  5]  13.00-14.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.91 MBytes
[  5]  14.00-15.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.91 MBytes
[  5]  15.00-16.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.91 MBytes
[  5]  16.00-17.00  sec  1.15 GBytes  9.90 Gbits/sec    1   1.91 MBytes
[  5]  17.00-18.00  sec  1.15 GBytes  9.91 Gbits/sec    0   1.91 MBytes
[  5]  18.00-19.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.91 MBytes
[  5]  19.00-20.00  sec  1.15 GBytes  9.90 Gbits/sec    0   1.91 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  23.1 GBytes  9.90 Gbits/sec   49             sender
[  5]   0.00-20.00  sec  23.1 GBytes  9.90 Gbits/sec                  receiver

Basically wirespeed.

So here it is when I transfer a 10 GB file AND write it to disk on the other side.

On nas-1:

root@nas-1[~]# iperf3  -F /root/10gfile.iperf2 -s

On vm-1:

root@vm-1[~]#  iperf3  -F /root/10gfile.iperf -t 20 -c nas-1
Connecting to host nas-1, port 5201
[  5] local 10.8.8.11 port 47482 connected to 10.8.8.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.08 GBytes  9.27 Gbits/sec   22   1.57 MBytes
[  5]   1.00-2.00   sec   962 MBytes  8.07 Gbits/sec    1   1.57 MBytes
[  5]   2.00-3.00   sec   995 MBytes  8.35 Gbits/sec    0   1.57 MBytes
[  5]   3.00-4.00   sec   680 MBytes  5.70 Gbits/sec    2   1.57 MBytes
[  5]   4.00-5.00   sec   405 MBytes  3.39 Gbits/sec    0   1.57 MBytes
[  5]   5.00-6.00   sec   389 MBytes  3.27 Gbits/sec    0   1.57 MBytes
[  5]   6.00-7.00   sec   388 MBytes  3.25 Gbits/sec    0   1.57 MBytes
[  5]   7.00-8.00   sec   380 MBytes  3.19 Gbits/sec    0   1.57 MBytes
[  5]   8.00-9.00   sec   372 MBytes  3.12 Gbits/sec    0   1.57 MBytes
[  5]   9.00-10.00  sec   415 MBytes  3.48 Gbits/sec    0   1.57 MBytes
[  5]  10.00-11.00  sec   401 MBytes  3.37 Gbits/sec    0   1.57 MBytes
[  5]  11.00-12.00  sec   349 MBytes  2.93 Gbits/sec    0   1.57 MBytes
[  5]  12.00-13.00  sec   354 MBytes  2.97 Gbits/sec    0   1.57 MBytes
[  5]  13.00-14.00  sec   365 MBytes  3.06 Gbits/sec    0   1.57 MBytes
[  5]  14.00-15.00  sec   365 MBytes  3.06 Gbits/sec    0   1.57 MBytes
[  5]  15.00-16.00  sec   365 MBytes  3.06 Gbits/sec    0   1.57 MBytes
[  5]  16.00-17.00  sec   382 MBytes  3.21 Gbits/sec    0   1.57 MBytes
[  5]  17.00-18.00  sec   379 MBytes  3.18 Gbits/sec    0   1.57 MBytes
[  5]  18.00-19.00  sec   319 MBytes  2.68 Gbits/sec    0   1.57 MBytes
[  5]  19.00-20.00  sec   299 MBytes  2.50 Gbits/sec    0   1.57 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  9.44 GBytes  4.06 Gbits/sec   25             sender
        Sent 9.44 GByte / 10.0 GByte (94%) of /root/10gfile.iperf
[  5]   0.00-20.00  sec  9.44 GBytes  4.05 Gbits/sec                  receiver

THAT’s the slowdown I was seeing, although not as bad as with ubuntu in the mix.

But it changes if I don’t write it to the disk on the iperf server side:

On nas-1:

root@nas-1[~]# iperf3 -s

On vm-1:

root@vm-1[~]#  iperf3  -F /root/10gfile.iperf -t 20 -c nas-1
Connecting to host nas-1, port 5201
[  5] local 10.8.8.11 port 46922 connected to 10.8.8.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   975 MBytes  8.17 Gbits/sec   45   1.56 MBytes
[  5]   1.00-2.00   sec  1.07 GBytes  9.22 Gbits/sec    2   1.56 MBytes
[  5]   2.00-3.00   sec   975 MBytes  8.18 Gbits/sec    0   1.56 MBytes
[  5]   3.00-4.00   sec  1.03 GBytes  8.85 Gbits/sec    0   1.61 MBytes
[  5]   4.00-5.00   sec  1.07 GBytes  9.21 Gbits/sec    0   1.61 MBytes
[  5]   5.00-6.00   sec   974 MBytes  8.16 Gbits/sec    1   1.61 MBytes
[  5]   6.00-7.00   sec   972 MBytes  8.16 Gbits/sec    2   1.61 MBytes
[  5]   7.00-8.00   sec   976 MBytes  8.18 Gbits/sec    0   1.61 MBytes
[  5]   8.00-9.00   sec  1.03 GBytes  8.85 Gbits/sec    0   1.61 MBytes
[  5]   9.00-9.95   sec  1.04 GBytes  9.34 Gbits/sec    1   1.61 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-9.95   sec  10.0 GBytes  8.63 Gbits/sec   51             sender
        Sent 10.0 GByte / 10.0 GByte (100%) of /root/10gfile.iperf
[  5]   0.00-9.95   sec  10.0 GBytes  8.63 Gbits/sec                  receiver

Almost wirespeed, but not quite I assume because it has to read from the disk on vm-1 (vm-1 has a 4TB SAS HDD mirror boot drive).

So, then I wanted to make sure that it wasnt the reading or writing, so I created ram disks on both machines and used that to read and write from.

mkdir /tmp/ramdisk
chmod 777 /tmp/ramdisk
mount -t tmpfs -o size=20G myramdisk /tmp/ramdisk

On nas-1:

 iperf3  -F /tmp/ramdisk/10gfile.iperf2 -s

On vm-1:

root@vm-1[~]#  iperf3  -F /tmp/ramdisk/10gfile.iperf -t 20 -c nas-1
Connecting to host nas-1, port 5201
[  5] local 10.8.8.11 port 41210 connected to 10.8.8.10 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.33 Gbits/sec   20   1.54 MBytes
[  5]   1.00-2.00   sec  1.12 GBytes  9.62 Gbits/sec    1   1.54 MBytes
[  5]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.56 MBytes
[  5]   3.00-4.00   sec  1.15 GBytes  9.90 Gbits/sec    5   1.56 MBytes
[  5]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.56 MBytes
[  5]   5.00-6.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.60 MBytes
[  5]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec    5   1.60 MBytes
[  5]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec    1   1.60 MBytes
[  5]   8.00-8.95   sec   899 MBytes  7.97 Gbits/sec    0   1.60 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-8.95   sec  10.0 GBytes  9.60 Gbits/sec   32             sender
        Sent 10.0 GByte / 10.0 GByte (100%) of /tmp/ramdisk/10gfile.iperf
[  5]   0.00-8.95   sec  10.0 GBytes  9.60 Gbits/sec                  receiver

So, basically a wirespeed transfer.

I think I’m happy with the networking now. At least it’s making sense.

2 Likes