Self-hosted Overlay Network Router for Globally-routable IP Addresses?

I am in the market for a self-hosted overlay network router product to transport globally-routable IP addresses, both IPv4 and IPv6. I do not know if such a product even exists. The product would have to operate in the environment described below. I am not sensitive to the form factor of the product. While I’d prefer VMs, I’ll take 19" rackmounts.

Colo Anchor Point and Future Gateway
At the colo, I have 2x legacy IPv4 /24 and 1x IPv6 /48 of globally-routable IP addresses.

Remote Office
At the remote office, I have a range of uplinks, one or all of which can be down at any given time. The uplinks range from DSL, to Starlink, to LTE. From a design perspective, some uplinks will support IPv4 only, some IPv6 only, and some both (poorly). All uplinks can be assumed to be behind a NAT, including the IPv6 uplinks. External IP addresses should be assumed to be changing frequently, with outgoing port numbers subject to change at any time and without notice.

Desired End State
A self-hosted overlay network that makes available at the remote office subnets of the /24s and /48. Isolating the remote office users from the ever-changing seething state of the physical links below it. Other than complete simultaneous failure of all uplinks of course.

Not Required
Any kind of firewalling, port filtering, or NAT. It would be nice if those were were included, but such capability can always be added by adding another router. This post here concerns itself solely with achieving physical transport of globally-routable IPv4 and IPv6 packets, however encapsulated, across changing and at times actively hostile underlying links.

Commercial suggestions welcome. If there were an open source solution, I suspect I would have found it by now.

Thanks,
– Lucky

If you’re looking for a mostly-point-and-click commercial solution, that would be “SDWAN without remote office DIA”. Meaning you get an SDWAN product (this takes care of building tunnels across every uplink combination between each endpoint) and you configure it to use the other end of the tunnels (the datacenter) as the default gateway for user traffic. In the SDWAN industry, “DIA” (Direct Internet Access) means allowing some traffic to leave the remote office via its local internet connections, instead of sending everything to the hub (datacenter / main office). Then at the datacenter you do whatever routing and NATing you want - usually this same SDWAN product will do this for you too.

The only small-scale option I can point to with some familiarity is Untangle SDWAN, but there are plenty of options. Just be sure to look a little bit beyond the marketing term of “SDWAN”. ZeroTier, for example, will help you build tunnels between locations/devices and provides what appears to be a direct ethernet switched connection between all of them, but it doesn’t have any of the path selection and uplink monitoring features that most “SDWAN” includes. Watchguard’s SDWAN is the opposite - it has a lot of intelligence for selecting which uplink to use for each type of traffic, but doesn’t do any site-to-site tunneling. SDWAN just means “Software Defined WAN” so there’s a lot of possible features that it might or might not include.

Since you’re just talking about two endpoints, (colo and remote office) implementing an SDWAN solution may be overkill. You can “easily” (in terms of complexity) set up the same yourself with PFSense, Mikrotik RouterOS, etc at each end. You manually set up tunnels/VPNs between the two (with the colo as the server since it has fixed IPs that aren’t NATed), and then configure your remote office to use a certain IP at the colo as its default gateway IP. Some protocols for tunnels are better at NAT traversal than others but it is completely doable.

Bruce,
Thank you so much for taking the time to write such a detailed answer!

Some follow-up questions are below:

If you’re looking for a mostly-point-and-click commercial solution, that would be “SDWAN without remote office DIA”.

I would tremendously appreciate names of manufacturers and specific products. Must be self-hosted, no cloud/MITM involved.

The only small-scale option I can point to with some familiarity is Untangle SDWAN

Having just looked at their datasheet, it appears that one would deploy a “Micro Edge” device per “remote office” an “NG Firewall” at HQ/the colo, and then there is a “Command Center”. Are those three different devices? What’s the rough aggregate cost of those devices, all-in?

Drilling down on the “Micro Edge” device’s wired WAN uplinks, the top tier features a “2x GbE / SFP Combo”. It is not entirely clear to me if that means that this enables the use of three separate WAN uplinks or if this is marketing speak for “two WAN uplinks and you get to pick the connectors”.

If there are only two uplinks, the device would not suffice for my needs. Three WANs would help dig me out of the hole that I am in today. But if I invest in hardware, I’d like at least four WANs, five would be better.

I totally understand your point regarding ZeroTier. It is a fantastic tool for what I use it for, which is to provide occasional networking support to non-techie friends: I drop off an old laptop with ZeroTier and a Rumble Agent on it and before long can fix whatever plagues them from the comfort of my couch.

WatchGuard I was unfamiliar with. You are correct that absent tunneling that’s not what I am looking for.

I looked long and repeatedly at OpenMPTCProuter and have nothing but respect for Ycarus. Still, OpenMPTCProuter is duct-taping together three completely different technologies to achieve its goals and in the end I’d still be left with an OpenWrt-based router. For me, that’s an unlikely path to happiness.

So what’s the next step up in number of WAN connections from Untangle?

(I do remain curious how much even an Untangle solution would cost for one remote site all-in and if that entails two or three WAN uplinks).

Big Thanks!
– Lucky

If you are going to establish tunnels from the office to the colo across each ISP, I would probably look to setup IPsec tunnels across the three ISPs and use BGP to determine availability. This can be done on cheap Cisco routers if needed. I would put the router in front of your firewalls and just let them handle the transport between your two sites.

What firewall/routers do you have in place now?

Untangle SDWAN and Untangle NGFW are unrelated products. Yes, they do often show them being used together - SDWAN bringing traffic into a datacenter or main office from remote offices, and then all the traffic to/from the unfiltered internet going through a separate NGFW. But their SDWAN product has sufficient routing and basic firewall chops that it probably suffices for you anyway.

Command Center is their cloud component that is included with the cost of any subscription (actually some features are available even for free with an unlicensed NGFW install). Aside from providing cloud-based remote access, the main purpose it has is to push the same policies out to a bunch of remote offices at once. Since you’re only talking about one remote office, I don’t think you’ll “need” to use it, but it will be a nice to have. Connecting to the cloud is not required for Untangle products to work, other than checking their license status. Command Center is not part of any network decisions, if you choose to use it to push down policies then that’s only helping you apply the same config to multiple devices simultaneously.

Untangle Micro Edge (which I didn’t realize they renamed their SDWAN product to until just now), just like Untangle NGFW, can actually be installed on any hardware of your choosing, you don’t have to pick one of their preconfigured appliances. Additionally, you aren’t limited to the physical ports on the device, you can connect more WANs to a switch, put each WAN in its own VLAN, and then use the VLANs as WAN interfaces.

It looks like they’ve dumbed down the “install it yourself” offering by not making a generic installation ISO available, and instead are offering a “Virtualization” option with premade VMWare images. So what you can do is purchase a generic server, install ESXi on it which has a free license option (totally sufficient for your needs) and then either pass through a bunch of NICs (preferably Intel) to the VM, or do what I suggested before with VLANs and make a bunch of virtual interfaces, one per VLAN.

I think I made one assumption that’s untrue now that I’m looking closer at it. I thought that Untangle Micro Edge did help automatically build tunnels, and provide a one-click “use X location as the gateway for all user traffic” option. You can definitely use Untangle Micro Edge to create tunnels and then make a routing policy to use those tunnels collectively as a gateway, but it will be more manual that I realized.

Assuming you do two virtualized MicroEdge installations (one at each end), and you don’t also have an Untangle NGFW install at the datacenter, you’re looking at $162x2=$324/year for 100Mb, or $238x2=$476/year for unlimited bandwidth. The license only cares about bandwidth, not number of WANs/tunnels/etc. All the pricing is publicly visible at Configurator | Untangle

The actual product names I know would be way, way overkill for you… Right now I’m working at a Fortune 500 and we’re using Versa Networks SDWAN. That product is competitive against offerings from Cisco and Fortinet. Meraki SDWAN would be a viable option, but that’s probably way too much cloud for you even though no traffic goes to the cloud.

Yeah I’ve looked at that project before for personal and nonprofit projects, but never got up the nerve to set it up. I’ve mostly gotten away with ZeroTier and/or EoIP tunnels, and some sort of routing on top of them. Its very much manual effort, but it is something that’s comfortable for me to set up and maintain.

This is really what all the big-name SDWAN products are doing in the background anyway - Versa, Cisco, Fortinet, Meraki. Its just IPSec tunnels and BGP. You can also do this with PFSense, Mikrotik, Ubiquiti EdgeRouter, etc.

2 Likes

One interesting protocol with Cisco to check out is PfR. It was probably one of the first to offer application performance based routing long before SDWAN was a thing. I’m pretty sure Viptela uses it. My one knock on SDWAN solutions is they generally cost more than the cost of higher bandwidth circuits. If you have enough bandwidth then why the need for SDWAN?