Ive been sick so never got in to the office until now, however i have run some iPerf3 testing with single and multi thread connections, and from what i understand of a 1Gbit connection what i’m seeing is about right. Maybe you can confirm.
I had a very similar experience on very similar hardware. Netgate (pfSense+) 6100 connected to a Unify 24-port POE Enterprise switch. Both Netgate and Unifi connected via 10G SFP+. My TCP download performance on a LAN-attached PC to the Netgate was line speed from my ISP (1.2GB). The same test run from the Unifi switch was always around 25% of the WAN download.
I ran many test. Swapped SPF+ DAC cables from one purchased from Netgate to one purchased from Unifi. Test were identical. I then experimented moving the pfSense–>Unifi downlink from SFP+ (DAC) to 2.5GB copper (RJ45) on another port. Same results.
The ONLY thing that restored my network to “normal” throughput everywhere was reducing the Netgate–>Unifi downlink to 1GB. This obviously eliminates any 10GB potential benefit to the Netgate/Unifi switch interface, but with all my PCs and other LAN devices at 1GB or 100MB, it is OK.
I read a great deal about pfSense Traffic Shaping, Switch Flow Control, TCP windowing, and other possible fixes, but they did not directly address the topic.
For the benefit of all, I am hopeful that someone with more experience may be able to shed light on the beat way to configure our LANs when we have slower WAN, to 10GB (higher speed) pfSense (router) and Unifi (switch), with 1GB or slower LAN connections (which for me include VLANs much like UK_TechDad described).
Tom or anyone else, can you suggest the optimal way to configure this pfSense/Unifi high-speed middle component with lower speed WAN and lower speed hosts? The performance bump in the middle (pfSense/Unifi) causes more harm than good without some throttling/tuning.
Thanks in advance for anyone’s help. UK_TechDad: I fully feel your pain after almost 2 weeks of my own trial-and-error.