I’ve got the following Setup:
Netgate 7100 with PFSense installed
FS S5850-24XMG L3 Switch
Zyxel XGS1250-12 Access Layer Switches
My biggest Problem at the Moment is that I’m not really able to get the 10G out of the PFSense Box.
For now I’m still routing all the traffic through the Firewall instead of handing that off to the L3 Switch because I’m not so comfortable with the handling of the Switch atm.
This also means that all the traffic has to go through the 1 10G Link between the 7100 and the FS Switch.
So before I’ve setup all my VLANS i tested the connection Speeds with IPerf3 on a default Configuration on PFSense.
With my Workstation as Client which has 2 hops to the Firewall (Zyxel > FS > PFSense)
I never got more than 1Gb on this Configuration altough all the negotiated Links show 10Gb negotiated Speeds.
When I test this in the other Direction, meaning the PFSense as Client and the Workstation as Server I get a little bit more out of the connection. About 1,8Gb per second to be clear.
I’m not sure if that’s because there’s something messed up at the Firewall itself or if it has to do with the wonky Fibre connection between the FS Switch and the Firewall.
Reason is the FS switch just has QSFP+ Uplink Ports and the 10G Ports on the Netgate 7100 are SFP+.
First of all I tried a 10GBase-T SFP+ Module on the 7100 and connected it to one of the 10GBase-T Ports on the FS Switch but Netgate already stated that this wouldn’t work because of driver Issues.
So I bought a QSFP+ to SFP+ Adapter from FS and FS allows to split up the Physical QSFP+ port into 4 Virtual Ports somehow…I did all of this with the Support of FS Staff.
From this Adapter configuration I plugged in a Fibre Cable and installed an Intel SFP+ Module in the 7100 to get the connection going.
Do you guys have any Ideas what I could check or try to find a solution to this Problem?
The only thing I don’t understand is why a 10gbps copper module wouldn’t work in the 7100, it’s the module doing the work, the interface shouldn’t care. But this thought is probably too simplistic as I know the modules can matter.
The only other thing I can question is how many vlans are running on that 10gbps connection, if more than 2 I’m wondering if pfsense is reserving bandwidth for the other vlans. Say you had 5 vlans, could it be reserving 2gbps each (more like 1.8 each with overhead)? I’m afraid I don’t know but maybe more things to think about.
If you check the Documentation on the Netgate 7100 optional Interfaces, which are some sort of Intel Nic I think, they clearly state that 10GBase-T modules are not supportet.
I also don’t understand why but i tried it and the connection just didn’t come to life
And regarding the Overhead, as I said I also tried it completely without Vlans in the beginning and it was no difference than now with about 7 Vlans lol.
But thanks for the answer anyway mate, maybe someone else knows the answer
What was the iperf commnad you ran? When it comes to the BSD kernel you wont be able to hit 10Gb on a single stream. Meaning you have to have more than 1 parallel stream to saturate 10Gb.
To test this you should append -P 10 to your command and rerun the test.
Alright just did this and it get’s a max of 1.95Gbit/sec
Is there a possibility to also enter this Option on the PFSense box to turn that around and make my Workstation the Server?
So the maximum I’ve seen so far were 3.18Gbit/s from the Workstation to the Netgate Box.
In the other direction it’s slightly under 2Gbit/s but I don’t know how to run parrallel streams on the PFSense Box.
Also when doing this Iperf Test the CPU Spikes to about 80%
The current norm is about 50% with 7 Vlans and PFBlockerNG installed.
But not a lot of devices currently online and talking on those Vlans.
I think you’re running into a limitation on traffic generation. If your cpu is hitting 80% and are only getting 3gig I suspect you’ll never be able to generate the 10g. I don’t remember the iperf command that allows you to control the frame size of the traffic but if you can increase it to 1518 byte frame (or make it jumbo) it should help lower the compute power needed to generate frames and it may allow you to increase the traffic rate - what size frames is iperf doing now?
tbh I don’t know which Framesize IPerf is currently doing…but tbh this is a Netgate Appliance I don’t get why it shouldn’t be able to do that with default settings
Anyway…I know for a fact that I can set the Frame Size on my L3 Switch to Jumbo and I think I already did that…but doesn’t that mean that Frames get more or less “repacked” at every device?
Thanks for the answer though…I will check that out as soon as I’ve got some sparetime!
I’ve set the MTU on the PFSense Interfaces now to 9000Bytes and also enabled Jumboframes on my L3 Switch but nothing changed…the CPU load still goes through the roof
I also tried to use “ping 10.95.1.1 -f -l 8500” to my pfsense now and it seems that nothing did change about the Packet sizes.
I have to go down to about 1465 until they don’t have to be fragmented anymore.
Don’t know…slowly I suspect my 10G Zyxel Access Switches to limit the max packet size and those got no settings for that.
I will have to test that with my PC directly attached to my L3 Switch, gonna need a pretty long cable for that I guess
But still…I don’t get why the PFSense shouldn’t be able to manage 10G with default 1500 MTU
I finally got to do it but I really think the Netgate Firewall reaches it’s limits there.
With 1 single IPerf stream I’m gettin about 4-5Gbit/s now and with 10 in Parallel I’m getting about 9ish now!
I also had to activate Jumbo Frames on the Nic on my PC then it started working!
OP, did you test the raw speed between your PC and the 7100 and what do you get?
Then from your workstation to the FS and to the 7100?
Then add the Zyxel?
Next, your fiber transceiver might not be compatible with the port they are connected too. That happens more than people think. FS seems to like Cisco type transceiver usually and has no problem with them.
On the 7100, I see from the photo you are not connecting your fibre to the IX0 or IX1 built-in 10GE SPF+ port, why? You are using an addon board on your Netgate right? And the 7100 is turning high CPU when there is traffic on those ports? Hmmm, I would move the fiber over the built-in ports as I suspect that PCIe 3.0 adapter might not be fully compatible.
First of all as you can see in my Photos, I’m getting those 10Gbit now…more or less.
I just had to use bigger packet sizes and this seems to have something to do with the CPU Load on the 7100.
This optional NIC on the 7100 you are referring isn’t just any old NIC…that’s an optional NIC which was installed by Netgate as an option at the Order…so it is compatible for sure.
Also…I’m using this Port because it is the Port on the outmost right side of the Appliance…and as the QSFP+ Ports on the FS Switch are also on the right side it makes Cable management cleaner than if i would have to route those to the complete opposite side.
Because of the FS Switch…I did ask the FS Support for a connection test from their QSFP+ Ports to the Netgate 7100 SFP+ Port and they did recommend the Transceivers to me that I have now installed.
I actually don’t know anymore what brand the transceiver on their side was but I know that it’s an Intel on the 7100 Side which Fits the built in Intel NIC…and the Transceiver on the FS side was chosen by the FS staff themselves.
Run your speed test for 60 sec.
The last photo seems to show that it was hard to reach ~8Gbps.
And you shouldn’t have to touch MTU as 1500 would not disrupt the speed but for a few Mbps.
Yes, redo the tests without the Zyxel please.
Well but it jumped from shy under 1Gbit/s up to about 4-5Gbit/s on one Iperf Stream as soon as I’ve setup Jumbo Frames on the PFSense Box, FS Switch and the NIC of my Workstation.
So the Framesize definitely makes a difference.
The Question is which device has the Problem with the Frame size…I personally still suspect the PFSense Box because the CPU Usage is still jumping up 30% as soon as I start the Iperf Test.
Testing it without the Zyxel Switches will take some time because I also have a few different Issues I have to clear → currently trying to set up the IPTV from my ISP which also seems to get tricky.
So I guess I the timeframe for testing it without the Zyxel Switches will be around 1-2 weeks.
