XCP-NG and TrueNAS Core 10Gb Speed Issue

Computer Hardware & Server Infrastructure Builds
1

I have recent installs of both XCP-NG and TrueNAS Core, both running on separate dual-processor ~ 20 core, 100’s of Gb of Ram each Dell (R730 series) Servers with SSD’s and 10Gb NIC interfaces. I’m testing VM server backups from XCP-NG to TrueNAS and seeing less than single gigabit speeds. When testing with iperf3 below, I am observing low speed with a single stream, but near 10gb speeds with 5 streams. What settings might I need to adjust on either XCP-NG or TrueNAS or both to achieve something in the multigigabit range for backup speed? TIA

(Single Stream)
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 31.9 MBytes 267 Mbits/sec
[ 4] 1.00-2.00 sec 31.9 MBytes 268 Mbits/sec
[ 4] 2.00-3.00 sec 30.5 MBytes 256 Mbits/sec
[ 4] 3.00-4.00 sec 31.5 MBytes 264 Mbits/sec
[ 4] 4.00-5.02 sec 30.9 MBytes 255 Mbits/sec
[ 4] 5.02-6.00 sec 28.9 MBytes 246 Mbits/sec
[ 4] 6.00-7.00 sec 30.4 MBytes 255 Mbits/sec
[ 4] 7.00-8.00 sec 31.1 MBytes 261 Mbits/sec
[ 4] 8.00-9.00 sec 29.6 MBytes 248 Mbits/sec
[ 4] 9.00-10.00 sec 28.0 MBytes 235 Mbits/sec

[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 305 MBytes 256 Mbits/sec sender
[ 4] 0.00-10.00 sec 304 MBytes 255 Mbits/sec receiver

(5 Streams)
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec sender
[ 4] 0.00-10.00 sec 1.65 GBytes 1.42 Gbits/sec receiver
[ 6] 0.00-10.00 sec 1.83 GBytes 1.57 Gbits/sec sender
[ 6] 0.00-10.00 sec 1.83 GBytes 1.57 Gbits/sec receiver
[ 8] 0.00-10.00 sec 1.80 GBytes 1.55 Gbits/sec sender
[ 8] 0.00-10.00 sec 1.80 GBytes 1.55 Gbits/sec receiver
[ 10] 0.00-10.00 sec 1.79 GBytes 1.54 Gbits/sec sender
[ 10] 0.00-10.00 sec 1.79 GBytes 1.54 Gbits/sec receiver
[ 12] 0.00-10.00 sec 2.75 GBytes 2.36 Gbits/sec sender
[ 12] 0.00-10.00 sec 2.75 GBytes 2.36 Gbits/sec receiver
[SUM] 0.00-10.00 sec 9.82 GBytes 8.43 Gbits/sec sender
[SUM] 0.00-10.00 sec 9.82 GBytes 8.43 Gbits/sec receiver

2

Do you have any jails running on TrueNAS?

3

No Jails. Pretty vanilla install.

4

Ran some additional testing and have isolated the issue to XCP-NG. Getting near full 10gb bandwidth from another server to TrueNAS Core.

5

Interesting issue. I’m running a similar same setup. I have not run iperf to test if I get the same results. But did you solve the issue?

6

No resolution yet. I’ve been playing with the MTU (9000 vs 1500) but that so far has not made any difference. More testing and adjusting one variable at a time is in order I guess.

7

Did you ever find the cause of your speed issue?

I am planning on rebuilding my home lab in the next few weeks and would like to avoid this (or fix it quickly) if possible.

I’m currently on ESXi and TrueNAS, moving to XCP-NG and TrueNAS Core.