Network Performance & Upgrades

Networking & Firewalls
1

I first want to thank @LTS_Tom for his recent video where he did some network testing with XCP-ng, pfsense and some virtual machines. I used his methodology to test my own network and here is what I found:

  1. VM to VM on the same hypervisor host was running at just above 6 Gbps
  2. VM to VM on the same hypervisor host on separate LANs going though pfsense was running at just above 1.2 Gbps
  3. Desktop to managed switch then to another to managed switch to pfsense to VM was running at just above 936 Mbps

This leads me to believe that I need to upgrade my hardware before looking at 10G networking.

My VMs currently run on either 1 of 3 HP DL380G5s or a Dell 2950G3. My switches are all D-Link DGS-1100 smart-managed switches and my Desktop is an older i7 with 16GB ram. All of my switches are connected via 4 LACP bonded ports. pfsense is connected to the main switch with a 4 port nic dedicated to the pfSense LAN interface and are using LCAP as well with 4 ports assigned.

for this test, I made 2 VMs on my Dell 2950 with 8 cores and 2GB ram and installed Ubuntu 18.04 and all the updates and then added iperf. My desktop is running PopOS 19.10 and has 2 NICs LACP bonded to the connected switch.

I am hoping to get some feedback on this testing that I have done and if I am drawing the correct conclusions as well as what upgrades should be made next and eventually I want to go full 10G networking.

2

I would say yes, your testing is producing expected results. LACP is a load sharing solution, not load balancing so an single TCP session will only be able to use one physical connection of a bonded pair.

I recently spoke of the specifics in this thread about how LACP works: Routing multiple network interfaces

3

My surprise was more with the VM to VM with and without passing the traffic through pfSense, though plan to run another test today as I remembered this morning when I made those 2 new VMs that i did not use local storage on the 2950 but the space on one of my NAS devices, I’ll see what the numbers are when I do the same test but have the VMs run local storage as pfsense already is.

I setup LACP both for redundancy but also as I typically have multiple devices connecting to multiple devices so LACP does help performace in that respect vs a lone 1G connection.

4

Is your pfSense virtualized or a physical appliance?

5

its is virtualized runing on a 2950 with 32GB ram and 2 Intel® Xeon® CPU X5460 @ 3.16GHz. the VM has 2GB ram and 8 cores assigned as well as a 10G HDD that is local to that machine. There are also 6 dedicated NIC ports via Linux Bridges. 1 for WAN1, 1 for WAN2 and 4 are trunked in the hypervisor and send to pfsense as a single NIC for the LAN. the WAN1 and WAN2 share a single dual port card and the LAN has a dedicated quad port card. I didnt do passthrough as my system does not support it.

6

Do your two systems running iPerf sit on the same virtual host?

7

Yes they are on the same host. Had 8 cores assigned and 2 GB ram. I did put the hdd on the storage NAS like my other VMs so maybe having them on the local disk like offense could help get a more accurate number