I first want to thank @LTS_Tom for his recent video where he did some network testing with XCP-ng, pfsense and some virtual machines. I used his methodology to test my own network and here is what I found:
- VM to VM on the same hypervisor host was running at just above 6 Gbps
- VM to VM on the same hypervisor host on separate LANs going though pfsense was running at just above 1.2 Gbps
- Desktop to managed switch then to another to managed switch to pfsense to VM was running at just above 936 Mbps
This leads me to believe that I need to upgrade my hardware before looking at 10G networking.
My VMs currently run on either 1 of 3 HP DL380G5s or a Dell 2950G3. My switches are all D-Link DGS-1100 smart-managed switches and my Desktop is an older i7 with 16GB ram. All of my switches are connected via 4 LACP bonded ports. pfsense is connected to the main switch with a 4 port nic dedicated to the pfSense LAN interface and are using LCAP as well with 4 ports assigned.
for this test, I made 2 VMs on my Dell 2950 with 8 cores and 2GB ram and installed Ubuntu 18.04 and all the updates and then added iperf. My desktop is running PopOS 19.10 and has 2 NICs LACP bonded to the connected switch.
I am hoping to get some feedback on this testing that I have done and if I am drawing the correct conclusions as well as what upgrades should be made next and eventually I want to go full 10G networking.
I would say yes, your testing is producing expected results. LACP is a load sharing solution, not load balancing so an single TCP session will only be able to use one physical connection of a bonded pair.
I recently spoke of the specifics in this thread about how LACP works: Routing multiple network interfaces
My surprise was more with the VM to VM with and without passing the traffic through pfSense, though plan to run another test today as I remembered this morning when I made those 2 new VMs that i did not use local storage on the 2950 but the space on one of my NAS devices, I’ll see what the numbers are when I do the same test but have the VMs run local storage as pfsense already is.
I setup LACP both for redundancy but also as I typically have multiple devices connecting to multiple devices so LACP does help performace in that respect vs a lone 1G connection.
Is your pfSense virtualized or a physical appliance?
its is virtualized runing on a 2950 with 32GB ram and 2 Intel® Xeon® CPU X5460 @ 3.16GHz. the VM has 2GB ram and 8 cores assigned as well as a 10G HDD that is local to that machine. There are also 6 dedicated NIC ports via Linux Bridges. 1 for WAN1, 1 for WAN2 and 4 are trunked in the hypervisor and send to pfsense as a single NIC for the LAN. the WAN1 and WAN2 share a single dual port card and the LAN has a dedicated quad port card. I didnt do passthrough as my system does not support it.
Do your two systems running iPerf sit on the same virtual host?
Yes they are on the same host. Had 8 cores assigned and 2 GB ram. I did put the hdd on the storage NAS like my other VMs so maybe having them on the local disk like offense could help get a more accurate number