VPN throughput dropped going to pfsense

Networking & Firewalls
1

Hi everyone,
Some background:
Site 1 and Site 2 used to be connected to each other over wireguard.
Site 1 is a protectli box and was running OPNsense. Prior to loading pfsense on it, I was running Wireguard on OPNsense and iPerf tests were giving me around 180Mbps. Site 2 is a 200Mbps line so this was perfect.

Site 1 is moved over to PFsense and now I cant go above 20Mbps throughput while on Wireguard. I made the changes in System > Advanced, and Enabled PowerD and set it to HiAdapative. Also enabled AE-NI CPU-based acceleration. Same settings I made when it was on OPNsense. No change to speeds.
I decided to change to IPsec just to test if maybe, somehow, the wireguard implementation was at fault. No change and in fact I get the same speeds - 20Mbps give or take.

To make sure I wasn’t crazy, I ran a speedtest.net test at Site 2 running OPNsense and the results are normal - 200/30. So I know its not trouble with the circuit. Same goes for Site 1. I get my 500/500.

The only change made was pfsense so my focus is on that being the item at fault. Anyone have any ideas ?

For what its worth I went through the following guide:
https://protectli.com/kb/pfsense-configuration-recommendations/

2

I’d try going through the these steps - Troubleshooting — Troubleshooting Low Interface Throughput | pfSense Documentation

It was a while ago, but i recall mucking around with the traffic shaper settings created a more stable and faster connection, but i don’t recall if i needed to do this for vpn throughput.

Hope this helps.

3

Do you have traffic shaping configured?

4

None. Here’s the weird thing. If I remote access using WireGuard or OpenVPN, I get great speeds (using internal speed test application). Just site 2 site that’s affected.

5

What mtu and mss settings do you have configured at each end?

It can help if mtu mss are the same and pfsense and opnsense may differ -
https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html

Wireguard info - vpn - Wireguard tunnel slow and intermittent - Super User

Note: not sure if you can edit opnsense mtu or mss, but you can set a fw rule as a work around from what I’ve read.
https://forum.opnsense.org/index.php?topic=23099.0;wap2

What encryption settings do you have at each end?

Start lowest encryption and see if that improves performance.

Hope this helps

6

Hey just wanted to update everyone here.
Dont think I’m facing an MTU issue as I’m able to send up to 1472 bytes over the tunnel.

ping -f 192.168.70.26 -l 1472

Pinging 192.168.70.26 with 1472 bytes of data:
Reply from 192.168.70.26: bytes=1472 time=39ms TTL=62

MSS clamping is set on the pfsense to 1400 bytes by default. Im honestly don’t know where the performance drop occurred but the strangest thing is that its only Site2Site that’s impacted. Remote Access coming in via OpenVPN or Wireguard I can download around 60Mbps.

7

I think the packets are being fragmented but the fragmentation is being hidden from you.

I don’t use wireguard on pfsense, but if your ethernet MTU is 1500, how could a full size packet go through the tunnel without fragmentation, since there is encapsulation overhead?

google clear don’t fragment bit vpn

You may need to look at what is happing at the ethernet interface level with tcpdump to know if the packets are being fragmented. Or perhaps there is another way. I am a pfsense neophyte, as I use an ER-X at home.

8

Possible dropped packets between the sites? Use mtr to find out.