Site-Site IPsec VPN throughtput single connection bandwidth limit?

Networking & Firewalls
1

Hello Everyone, I currently have a site to site vpn setup using IPsec that I can not figure out why connections are limited to about ~20Mbit in both directions when they are a single connection. But if I run a parallel iperf test I can saturate the full download speed of Site B. Is anyone aware of a bandwidth limit per connection on IPsec vpn’s?

Background info
Site A
1000/1000 Fiber
Pfsense Firewall

Site B
200/20 cable
Sonicwall

VPN Settings
Phase 1 AES 256bit SHA256 DH5
Phase 2 Protocol ESP AES 256bit SHA256 PFS key group 5

SiteA-B_10connection
SiteA-B_10connection699×408 29.5 KB

2

IPsec is limited by the speed of the hardware Netgate has the performance of this systems listed here:
https://docs.netgate.com/pfsense/en/latest/book/hardware/hardware-sizing-guidance.html

3

Thanks Tom!

I took a look at that article and I think the issue is likely the states table (400000). I figured with the cpu power for the pfsense box running a Xeon (4threads) it should not matter, but now I think it’s kneecapped by its’ ram capacity (4gb).

You wouldn’t have any insight into why iperf3 would still be able to overcome the limits when 10 parallel connections are ran in test a compared to a test on a single connection?

4

One thing to consider is latency and TCP windowing. When you run multiple threads/streams you will get higher throughput as long as you don’t saturate your internet connection. As you pointed out, your lowest ceiling for sending traffic to Site B is 200Mbit. The slowest part of your communication is the ACK back to the sender. If latency is high, it will affect your throughput since it has to wait longer for the ACK. When you run multiple streams (parallel connections) this process is likely staggered and therefore you get near full throughput.

5

Thanks Fred

Latency between the sites is pretty jittery and higher than I would like. 50-110ms. Maybe some day site b will get fiber.

Thanks again

6

Latency has more to do with the distance between sites than anything else so regardless of what provider or type of circuit you get it won’t really change.

7

Absolutely, as well as the design of the isp’s network and peering agreements. Such as in this case the locations are only separated by 10 miles as the crow flies, but the network design of the cable isp causes the data to travel 300+ miles in this case.

1 Like