10Gb link showing only 5Gb transfer speed

Networking & Firewalls
1

Hi Folks

Need some help troubleshooting 10Gbps link speed. I have 3 devices (PFSENSE, TRUENAS and WINDOWS 10 PC) all three are connected to Mikrotik CRS328 switch on the SFP+ ports using 3m DAC cables. The PFSENSE and TRUENAS has Intel x520-Da2 cards in them. The Windows 10 machine has a Asus XG-C100C card. I am doing a speed test between these machines using iperf3 and I am not getting the expected values (hoping to see around 9Gbps atleast). Can you suggest on some troubleshooting steps to identify the issue? The summary of iperf3 tests between the machines is given below

Client 	--  Server
==============================================================================
Win10    -- PFSENSE
==============================================================================
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-20.00  sec  12.4 GBytes  5.31 Gbits/sec                  sender
[  5]   0.00-20.00  sec  12.4 GBytes  5.31 Gbits/sec                  receiver


=============================================================================
Win10    -- TRUENAS
==============================================================================
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-20.00  sec  20.7 GBytes  8.91 Gbits/sec                  sender
[  5]   0.00-20.00  sec  20.7 GBytes  8.91 Gbits/sec                  receiver

==============================================================================
TRUENAS  -- WIN10
==============================================================================
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.15 GBytes   925 Mbits/sec    0             sender
[  5]   0.00-20.12  sec  2.15 GBytes   919 Mbits/sec                  receiver

==============================================================================
PFSENSE	 -- WIN10
==============================================================================
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  2.19 GBytes   940 Mbits/sec  141             sender
[  5]   0.00-20.11  sec  2.19 GBytes   934 Mbits/sec                  receiver

==============================================================================
TRUENAS  -- PFSENSE
==============================================================================
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  12.1 GBytes  5.21 Gbits/sec    0             sender
[  5]   0.00-20.00  sec  12.1 GBytes  5.21 Gbits/sec                  receiver

==============================================================================
PFSENSE	 -- TRUENAS
==============================================================================
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  14.7 GBytes  6.33 Gbits/sec    0             sender
[  5]   0.00-20.00  sec  14.7 GBytes  6.33 Gbits/sec                  receiver
2

Ok so i narrowed down the issues to 2 problems

  • PFSENSE is not able to send receive higher than 5Gbps
  • Windows 10 is not able to receive greater than 1Gbps

I have identified the cause of the first issue. I am using a Intel Core i5-6500 CPU @ 3.20GHz in PFSENSE. When i run iperf3 with 1 thread, one core of the CPU reaches 98% utilization and iperf3 thruput maxes out at 5Gbps. If I run iperf3 with 4 threads then each core reaches about 80% and the iperf3 thruput reached 9.3Gbps.

So to resolve this does it mean that i have to upgrade the processor ??
I also realised in PFSENSE TCP offloading has been disabled. Can i enable TCP offloading to improve thruput ? Will it impact the functionality of PFSENSE in anyway?

I still have not been able to identify the issue why the WIN10 machine wont receive data higher than 1Gbps. I disabled the AntiVirus, but no improvement.

3

Maybe a stupid question. But are you sure your cable is correctly plugged in to Asus XG-C100C? Or that not two cables are connected to the PC? And if so, that your router (or Windows 10) is reporting that the connection is a 10 Gbps one?

4

Thanks for the response.
Yes the cable is connected to correctly and i know the card works because I am able to get near 10GB transfer rate from WIN10 PC to TRUENAS Server. It is the reverse traffic that is the problem

In windows 10 the network is showing as 10Gbps.

5

Just checking I understand (rounding the numbers off a bit);

10 - pfSense = 6g
10 - TrueNas = 9g
pfsense - 10 = 1g
pfsense - TrueNas = 6g
TrueNas - pfSense = 6g
TrueNas - 10 = 1g

Assuming that the 10-pfsense, pfsense - Truenas and Truenas-pfsense are all now closer to 10g if you multi thread then it looks like something on the receive side of the Win 10 card. I realise that’s not a solution but it might help with debugging.

Maybe google “Asus XG-C100C receive buffer” or " Asus XG-C100C only 1g inbound"

6

on my windows 10 (also Asus XG) when I copy a 15 GB file from my unraid server to my SATA disk I get around 90 MB/s (720 Mbps). But when I copy it to my NVMe disk I get 700 MB/s (5,6 Gbps). In my case the unraid server is the problem. Maybe check your hard disk in your Windows 10 system…

7

Well i am one step closer to solving all issues. on the WIN10 machine i changed the following settings in the ASUS card

IPV4 Checksum Offload - Disabled
Large Send Offload V1 and V2 - Disabled
Receive Buffer - 2048
Transmit Buffer - 4096

After doing this tested all cases again and the following is the result

Client 	--  Server
==============================================================================
Win10    -- PFSENSE (Single or Multithread makes no difference)
==============================================================================
[SUM]   0.00-40.00  sec  20.2 GBytes  4.34 Gbits/sec                  sender
[SUM]   0.00-40.00  sec  20.2 GBytes  4.34 Gbits/sec                  receiver

=============================================================================
Win10    -- TRUENAS
==============================================================================
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-40.00  sec  40.0 GBytes  8.59 Gbits/sec                  sender
[  5]   0.00-40.00  sec  40.0 GBytes  8.59 Gbits/sec                  receiver

==============================================================================
TRUENAS  -- WIN10
==============================================================================
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-40.00  sec  42.9 GBytes  9.22 Gbits/sec    0             sender
[  5]   0.00-40.11  sec  43.0 GBytes  9.22 Gbits/sec                  receiver

==============================================================================
PFSENSE	 -- WIN10
==============================================================================
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-40.00  sec  42.2 GBytes  9.07 Gbits/sec    0             sender
[  5]   0.00-40.10  sec  42.4 GBytes  9.07 Gbits/sec                  receiver

==============================================================================
TRUENAS  -- PFSENSE (Multiple Thread) 
==============================================================================
[SUM]   0.00-40.00  sec  42.3 GBytes  9.09 Gbits/sec  288             sender
[SUM]   0.00-40.01  sec  42.3 GBytes  9.09 Gbits/sec                  receiver

==============================================================================
PFSENSE	 -- TRUENAS  (Multiple Thread)
==============================================================================
[ ID] Interval           Transfer     Bitrate         Retr
[SUM]   0.00-40.00  sec  37.4 GBytes  8.03 Gbits/sec    0             sender
[SUM]   0.00-40.00  sec  37.4 GBytes  8.03 Gbits/sec                  receiver

Now looks like all problems solved except for WIN10 - PFSENSE. It seems stuck at 5Gpbs. Even running parallel threads makes not difference. Infact if i run parallel threads the thruput drops marginally to 4.7Gbps.

8

I think at this point I’d chalk that up to “strange but probably not going to actually affect anything so I’ll leave it”… You could spend days trying to work out what’s up but in reality are you ever going to need >5gbps across the pfSense box?

1 Like
9

Couldnt agree more…definitely dont see a usecase where i can pump more than 5Gps across pfsense, but then its like that itch you cant scratch… keeps bothering me… why cant it reach that magic number…:slight_smile: