Hello. Would like to get some guidance on this project of mine and see if I may be on the right track.
We have a situation here where we will have two physical router gateways on the network, each connected to its own ISP. Due to business requirements, we cannot merge the two WANs onto a single router, thus two routers must exist.
Going forward we want users to be able to VPN into either gateway of their chosing, and be able to access the internal-web-server, as seen in the diagram below.
The general idea below is to setup a dedicated HAProxy server, set all devices’s default gateway to point to the HAProxy, and have HAProxy handle the connections to the routers. Unfortunately I am quite new to this whole proxy thing and not sure if HAProxy would do what Im looking for. Or can I place a pfSense router with HAProxy package on there and achieve the same thing?
As a FYI, currently we have all devices’s gateway are pointing to Router Gateway 1. The problem is that when remote users VPN via WAN-XYZ, into Router Gateway 2, they are unable to access the internal-web-server. To my understanding, this problem is due to the default gateway on the internal-web-server is set to Router Gateway 1.
I don’t understand how business requirements could drive a overly complex setup with two separate routers but one work around would be setting up both routers as gateways on each system which means setting every system as static IP and choosing one of the gateways as default.
To give some background. Router 1 is a cisco managed service router thus we have no control over it. While Router 2 is an old Netgear router, though soon to be replaced with pfSense, acting as a backup connection. This idea in the past was that if Router 1 fails, or connections from Router 1 becomes unavailable, we will manually update all default gateway to Router 2. However as the number of static ip devices grew, this process became quite time consuming, and a bit of a pain as some services will need to be ‘restarted’ to have the new gateway to take effect.
And in addition, as service cost rises, the business want flexibility in its fibre service providers and/or managed service providers going forward. Thus I was thinking if the proposed setup might provide this flexibility. The idea is that new fibre services can be directly connected to Router 2/pfSense, configured as normal. And if new service providers were to come on board, then, their routers could be simply attached to the main switch, and we simply update the proxy server while all devices on the network remains untouched. No special configuration required on their end or ours, except updating our proxy server with the new router details. And remote users can continue to log in without disruption.
I was also thinking of reorganising everything down to one pfSense router, or run HA with pfSense to provide redundancy. However, this means that if new managed service providers were to come on board, then special configuration will be required. And troubleshooting in the future may be more complicated, though this proposed approach also have its own complications.
But yes, because we may have multiple fibre service in the future and at the same time also possible managed service providers in the future, thus wondering if this setup might work.
Are there actual reasons you can’t feed both internet connections into pfsense and create a dual WAN connection? You don’t need control over the ISP gateway to do this…
It’s a very common setup, most companies have 1 major ISP and 1 backup that may be slower from another ISP. Sometimes this even a mobile network (desperate times desperate measures).
What you’re describing with HA proxy is effectively the same result as a dual wan pfsense. Except using a load balancer / reverse proxy as a gateway/router is more difficult in setting up and supporting than it would be to just setup pfsense in a dual wan config with traffic shaping.
As far as external clients go, you can have multiple IP address associated to a DNS name. Create two A records, often clients will automatically toggle between the two when re-attempting connections. This is called round robin DNS. This isn’t the “best” solution but a free one. The better solution would be hosting a load balancer on something like linode to direct clients based on health checks.
This was my original plan. However the fibre service on the managed service router, is also provided by the router service vendor. I had enquired on the necessary configuration, eg vlan 10 ppoe vs dhcp, they advised they cannot do this, as all connection to their service must be via their managed routers, fair enough.
Or, did you mean connect both ISP gateway to one pfSense router? From my understand with this approach, the ISP gateway router will need to be re-configued with some sort of port fowarding or DMZ so that all traffic are forwarded through. Because this approach requires them to reconfig the existing router, I worry that if something isnt configured properly, we may face some undesired outages.
And in addition, we also host a offsite backup nas in their data centre. They advised we will not be able to access the offsite backup without their managed router. Hence we cannot replace their router.
And yes, I do agree, adding HAProxy seems to be over complicating the issue especially its doing what a router/pfSense is doing. However, I am unsure, without some sort of proxy, how I could get remote users into the network via either gateway/ISP and access all network resources, and yet without reconfiguring all the devices on the network for both gateways. I also feel I am missing something. Hence I havnt implement this setup and still reviewing if there is a better approach.
And re the multiple IP with DNS name. Thank you. That is a clever little approach! Noted
If connecting to the ONT isn’t something they support then there’s no harm in connecting pfsense to their gateways. You don’t need to do anything on their gateways and it wouldn’t change the way anything is today anyway.
Based on what you’re saying, I assume their gateways are NAT’d and thus you can’t assign the public IP to your pfsense? Some ISP’s who want their equipment on-site still allow you to assign the public IP (assuming it’s static) to your own router.
Assuming they are NAT’d, then it’s not really a big deal. Whether you have pfsense inbetween them or not doesn’t change the way you operate. Without pfsense how do you forward ports you need? However you do that, doesn’t change except you’d now add it to pfSense (in addition to however you already do it).
pfSense would simply have a private IP for its WAN addresses, which is totally fine - there’s a setting you just need to uncheck (block private IP ranges) on the WAN interfaces and away you go.
Re the double NAT, yes this was what I was hoping to avoid. Because VPNs are in involved, I wasnt sure how doube NAT could affect this, though best practice seem to advise to avoid it. However the alternative option with HAProxy have its own issues…
Going with the double NAT option, diagram below. If my understanding is correct, I would at most need to request a static route to be added onto the managed service router (Router Gate 1). Assuming remote users are coming in via Router Gateway 1 and wanting access to the internal web server.
As a FYI, I setup a test environment mimicking the diagram from above. Used two pfSense routers and vpn via OpenVPN, worked better than I had expected! I think I will go down this route! Thank you Mikensan!
Though with the VPN, I think because I had tested it with OpenVPN, no issues were encountered. I know with the managed service router, they are using IPSec. Reading around forums, seems to be a potential issue. Is it just a matter of forwarding some IpSec ports to get VPN going in a double NAT situation?
Sorry for the late reply - alternatively you can setup linode (or any other cloud provider) to act as a proxy VPN. So your network dials-out to Linode, and then all your clients dial in to Linode. Linode’s smallest plan allows 1TB of traffic a month I believe ($5/m).
There’s similar solutions with cloudflare as well which is free - but I haven’t personally tried this (yet).
This way you don’t need any open / forwarded ports.
I missed your question in my response - you shouldn’t need to ask for a static route if the ISP1 gateway hands Router Gateway 1 an address - it will automatically add it to its routing table.
You would though request any port forwards you think you may need. Or if the ISP is willing to put their edge router in a bridge mode or 1:1 NAT to pass the IP Address over to your gateway, would be great.
No worries about the delay. Am already grateful for your time and advice!
And thank you for the VPN Proxy idea. Have heard about this approach in the past but never tried it myself. Always thought it was a complicated type of setup. But over the last week or two, digging heavily in VPN configuration and setup, I think I can give this a try now!
And thank you for the cloudflare guide. That is quite an interesting approach. Will give that a try in the near future. Quite a brilliant idea really!
As a FYI, I’ve ended up implementing the following design. Its actually more or less towards the original thought process. I’ve simply added a PFSense router onto the LAN, aka proxy gateway, and had all the LAN device gateway set to this proxy gateway. Proxy gateway was configured with the firewall service disabled, NAT disabled and added a static route for each VPN tunnel so their packets can directed back to their respective router/source. By doing this, the remote team can VPN in via either Router 1 or Route 2, and still able to access all the services on the local LAN.
The biggest issue I see with this implementation is:
This creates an asymmetric routing situation. I know this is often frown upon.
Outgoing internet traffic must traverse into and out the proxy gateway on same network interface. Where the throughput on that interface would be halved, and once significant traffic is reached, no doubt poor throughput rate will be encountered. Though Im wondering if there is some way this can be mitigated.
Benefits:
Router 2, existing managed router and its configuration, can remain exactly as is.
Modularity of managed service providers. Adding, transitioning or removing managed service providers can be done with minimal special configuration as network traffic can be easily redirected in the proxy gateway.
Redundancy in hardware failure without HA setup, and ease of traffic redirection. Any managed service router can physically fail, and the network traffic can be redirected in the proxy gateway either manually or automatically via PFSense auto failover ability.
But yeah, this is what I had implemented, seem to satisfy what I was looking for earlier.
Though open to any suggestions or flaws that can be seem with this approach, Im sure there are some factors that I have not yet considered or have overlooked.
