I want to replace public cloud services like google with my own small cloud system at home. Amongst these services will be some things that require semi high availabillity (SHA). “Semi” in that a few seconds or minutes of downtime is not an issue.
I’m not sure yet to what extend I want/need to expose some services to the internet, but in any case I want a decent firewall to secure the whole thing, and I think pfsense is a suitable candidate.
SHA should cover both, a primary WAN failure, as well as a hardware failure. Hence, my idea was to have a proxmox HA cluster with (at least) three nodes. The first node is connected to the primary fiber WAN modem, and a 5G USB modem is connected to the second node.
The idea is that if the first node fails, proxmox will migrate the firewall to the second node. At startup, pfsense detects the different hardware configuration and establishes a WAN connection over the 5G modem. When the first node is restored, the whole thing goes the other way…
Being new to pfsense, I have no idea if it supports such a system and would like to ask the experts here in the forum.
As a sidenote: I can’t run pfsense HA because my ISP doesn’t support CARP-VIP (or doesn’t want to in a consumer plan). Hence the idea of having pfsense virtualised and thus a semi HA system.
PS: If the underlying problem should be tackled in a completely different way, please let me know! I’m open to any and all solutions.
I’ve never done this, but I find the idea intriguing. I second Tom’s suggestion of making the WAN connections available to all cluster nodes. This can be done by connecting the modems to a switch, each in a dedicated VLAN, and connecting the firewall VM to these VLANs through Proxmox. Of course this will add the switch as a single point of failure, but I think at the level we’re talking about that is an acceptable compromise. If your modem has multiple Ethernet ports, you might even be able to use redundant switches.
On the one hand, I want to build an MVP, but on the other hand, I don’t want it to have a SPOF. Since the fiber network is connected to the first node and the cellular network is connected to the second node, there is no SPOF in this setup. So for now, I would like to assume that each node is connected to only one WAN device. Thus the original question, can pfsense handle different WAN devices at boot time and can it perform different dial-up procedures depending on which device is available?
I already have different scenarios in mind on how to improve the system later.
What Tom and Paolo described for the fiber side, for example if the router/modem supports brindging to multiple LAN ports. For the cellular side, I want to use a cheap USB modem. But connected on the second node, and passed through to a guest that acts as a cellular-to-LAN bridge. That way pfsense can use this LAN as a “failover WAN” regardless of which node it’s running on. And so on.
But that is all stuff for a later phase, when I have more time.