I’ve done this with Netscreen (ScreenOS), and with Fortigate, but wondering if pfsense had the concept of duplicate interfaces, or interface groups, essentially allowing it to work through a full mesh scenario. (allowing the pfsense to talk to 2 different switches but as 1 interface, where the election is done internally). Pasting a picture…
Thanks all!
-Chris
Lookup CARP. It is basically another option to VRRP or HSRP that other network equipment vendors support.
pfsense supports HA and I have a video on how to set it up right here 
The HA does most of the job, but there is a thread I’m trying to pull with regards to redundancy…
The issue that I see with this (if you look at a non meshed setup.) Assume you have 2 firewalls, and 2 switches at the tier below them. If they only have a connection to one downstream switch you could find yourself in a situation where the “Bottom left switch, and the Upper right pfsense” go out and then it would cause an outage as there would not be a path out for the server connected to both switches to go out.
Like this image…
VS a full mesh… I’m just not aware that CARP can overcome this particular situation…
Using CARP on the firewalls is still required for HA, but I can see where you are going with this. As you move down the stack in connectivity, this being the switches, there are two basic options for redundancy.
Since there is no connectivity between the switches in your diagram I’ll assuming that you can’t run vpc or lacp across the pair, so you will have to use a dynamic routing protocol such as OSPF or BGP.
Basically you need to setup L3 connectivity between the server links to the switch links, run BGP or OSPF across them and then statically route to the VIP of the firewalls. Personally if I am using routing protocols in any part of the network I would try to configure it end to end.
I apologize, those should have been linked together. The problem is that without the redundant interface (at least what I know them as), when you lose opposite sides of the chain you get an outage. I’ll go revise the images though for clarity.
Switches in my case are Force10 (now Dell) S4810 but assume no Cisco Nexus style capability.
Let me know what model of switches too please.
Going through the pfsense blog another has asked this
But it sends them here…
https://docs.netgate.com/pfsense/en/latest/book/vlan/index.html
But that doesn’t seem to mention virtual interfaces (or redundant interfaces)
Essentially 2 ports that sit / act as 1 port, it lets you make the “X” part of the full mesh. JunOS (also a FreeBSD variant) didn’t allow it, but I thought I had heard that PFSense did.
Based on a quick search it looks like those switches run Virtual Link Trunking (VLT) which is similar to vPC on a Cisco Nexus switch. This allows you to run a port-channel or LAG across switches from a single source (your server).
I would recommend you configure VLT on the switches to connect to the server via LACP. The server will than have an IP that uses the CARP IP as it’s default gateway. The nice thing about this design is that anything is layer 2 between the CARP IP and the server, which for most people is less complex.
That part I have down, but the issue is this… say you lose the left switch as in the diagram, and the right pfsense… you have no physical path right?
Yes, no matter what, you have no connectivity, but this is an N+1 design so you are only ever expecting a single device to fail. The only scenario that I see two devices going down at the same time is if they have a single PSU and opposite devices are plugged into the same PDU. If you stack the power to the upstream device on the same side you are fine.
Like I said, just wondering if anyone had experience with getting pfsense to actually do a full mesh (redundant interface, virtual interface, something like that.) If not no biggie I can just use something else but after watching the vids you all are definitely super knowledgeable about this product, thought I’d ask the masters 
On a fortigate cluster you would do something like this…
Just an image I had from an old install of the “left” fortigate… port 13 goes to the left switch, port 14 the right switch… the “right” fortigate does the same thing with its 13/14 and that creates the X.
That said, these fortigates out budget a lot of projects and I prefer supporting open source vendors.
you’re basically asking “can I run CARP on top of LACP?” (can I run a multi-router redundancy protocol on top of a multi-link redundancy protocol) - because AFAIK that’s what’s happening with your fortigate example. I don’t know the answer to this question, but oftentimes knowing the proper question to ask is helpful for finding the answer. Hopefully you will find that it is possible to set up CARP on top of LACP, but maybe PFSense isn’t architected for that.
Not exactly, those switches don’t have those ports in LACP, they’re just standard trunk ports. In reality I think I’m asking can I build a virtual interface that is failover tolerant (meaning only 1 is up at a time so it doesn’t confuse the down stream switches)
PfSense does allow you to bridge multiple ports together and give them a single ip. I’ve had a quick look at it appears you can then bind a virtual carp ip to that bridged interface. I’ve no idea if it works or how much of a performance hit you will get bridging two ports. Having said that I do actually have two ports bridged (I needed an extra one for my IoT network and didn’t want to buy another switch), and I’ve not seen any performance issues. Of course home IoT use and enterprise LAN use are two different things! Does that help?
