I know the basics but I am trying to get a deeper understanding of how they work and how I can improve upon my setup.
So here’s the back-story which is relevant just so that you know how my current setup is…
It all started with me getting into self-hosting. Next thing I know I had 19 different URLs (nextcloud, bitwarden, emby, IPMI etc etc.) that I had to remember the IPs and the ports. So I thought of getting a reverse-proxy. Obvious choices, Apache & Nginx. But then I thought, why not get proper SSL Certs from Lets-Encrypt along with the reverse proxy so that it avoids the browsers from screaming about it and my wife calling me over whenever she is accessing bitwarden or emby etc.
Enter Caddy2 which had easy integration with Lets-Encrypt DNS challenges. I needed the easy button because this was all very new to me. I bought a domain name for myself from Namecheap. Unfortunately, Caddy2 only had the Cloudflare plugin available for DNS challenges during 2.0 Beta. So, I created a Cloudflare account and then used those nameservers as my “Custom DNS” in the Namecheap account instead of using the Namecheap BasicDNS.
I set up 19 different A records – all pointing to my public WAN address (say XX:XX:XX:252) and using Cloudflare as the Proxy. I used DNS challenge and everything works as expected. I can use the sub-domains I defined in the A records instead of remembering the IPs and ports.
I use Opnsense as my firewall. I also have a road-warrior VPN server that I connect to from the road. I also have a dynamically assigned IP address. If my WAN IP changes, I would still want my certs and my VPN to continue functioning. Enter DDNS. I enabled DDNS service in Opnsense, and used the Namecheap option – put in my domain name (that I had purchased), my user/password and it immediately listed my WAN IP (XX:XX:XX:252) as the Cached IP. So here’s where I am confused
How did it cache my WAN IP for my domain name instead of the actual public IP of the domain name? In Cloudflare, my base domain points to a completely different IP (I am not hosting anything on that domain though)
Then my WAN IP changed when I rebooted the modem and the Opnsense firewall
However, all the A records that I created for the 19 services still point to the old WAN IP address (XX:XX:XX:252). This will be a problem whenever my current LE certs expire, wouldn’t it? Is there a way to auto-update these records whenever my WAN IP changes?
Is there a way to create a wildcard cert for my domain name so that I can use the same cert for all my LAN services?
How do I use the DDNS service in Opnsense such that my WAN IP is always tied to a particular domain name that I can use for all my VPN clients – so that I don’t have to manually change the IP address in each client’s VPN config?
So, to confirm, you’re using Namecheap’s DNS but your Namespace is hosted in Cloudflare?
If so, the way you’d do this is to use Cloudflare’s DDNS, as they’re hosting your Namespace.
How did it cache my WAN IP for my domain name instead of the actual public IP of the domain name? In Cloudflare, my base domain points to a completely different IP (I am not hosting anything on that domain though)
This occured because OPNsense knows your public IP, so it will inform your DDNS providor of the IP and update it periodically in the event of a change. It uses the information you gave it with your username and password. Each service is a bit different.
However, all the A records that I created for the 19 services still point to the old WAN IP address (XX:XX:XX:252). This will be a problem whenever my current LE certs expire, wouldn’t it? Is there a way to auto-update these records whenever my WAN IP changes?
Yes, you updated your A record on Namecheap but not Cloudflare where you host it.
Is there a way to create a wildcard cert for my domain name so that I can use the same cert for all my LAN services?
Look at the ACME plugin with OPNsense, it will create a wildcart cert you can use with NGINX or HAproxy
How do I use the DDNS service in Opnsense such that my WAN IP is always tied to a particular domain name that I can use for all my VPN clients – so that I don’t have to manually change the IP address in each client’s VPN config?
Create a DNS entry for something like VPN.yourdomain.com. Or, you could use literally any other of your DDNS records as they resolve to the same IP.
Ideally you’d want to create a DDNS entry in OPNsense for every subdomain you’re using that way all of your DDNS subdomains are up-to-date.
Additionally, are you securing these services you’re exposing the the internet? That’s a pretty big threat surface to have wide open.
Not entirely sure of the difference between DNS and Namespace, but I will look it up and come back with an answer to your question
Got it. Because I used the second-level & top-level domain name only, it set my WAN IP to that domain.
I was hoping to keep all the certs maintained from my reverseproxy (caddy2). I will look into creating a wildcard cert via caddy2.
Understood. It’s starting to make sense to me now…
I see. I will try to create a DDNS service for a few of my sub-domains and see if it works.
None of my services are available over the internet. I am creating these A records only to be able to get a Lets Encrypt cert for those sub-domains. The only way to get to these services would be if you were on the LAN or connected via the VPN Server that I have set up in Opnsense. Would there still be a risk?
OK so this changes a load. You can accomplish this super easily by using ACME and HAproxy / NGINX on your OPNsense router. I do the same thing at home. You don’t need to do any external DNS entries for your services. There’s no reason to add these A records to public DNS. I’m not entirely sure how Caddy2 handles LE certs, but if able to automatically swap them out when they’re renewed it might work.
To circumvent making pubic DNS entries you can simply add a DNS override for your services via OPNsense if you’re using that for DNS.
Additionally, I meant “Nameserver” not “Namespace” earlier. You can purchase a domain through one provider and move the nameservers to a DNS provider.
Honestly, I thought that using a separate reverse-proxy would be simpler in terms of certs management. But it seems I could have been mistaken. I might have to look into this. Is this same as Services → Web Proxy option in OPNsense? Or is this a separate HAProxy plugin?
I do have all the sub-domains set up as host overrides in Unbound resolver (which is what I use for DNS)
Yes. So that is the exact setup I have. I purchased the domain from Namecheap, but I am using Cloudflare Nameservers (only because I was using caddy2 and it didn’t have a Namecheap plugin at the time)
I am thinking of moving everything to Cloudflare from Namecheap – just to reduce complexity – and then also look into the HAProxy option that you mentioned. If you have any particular links which can explain HAProxy setup in OPNsense which can also be able to get wildcard DNS challenge Let’s Encrypt certs for the LAN services to a NOOB, I would appreciate it.
I’d highly recommend cloudflare for DNS, you can leave the domain at namecheap and just use cloudflare for DNS without issues. You’d only have to interact with namecheap for renewals. Most of my domains are bought through Google, but I do have some with namecheap.
Thanks @gsrfan01 for the links. I will look through and see if I can switch over to using HAProxy.
You had mentioned in a previous post that I wouldn’t need any external DNS entries if I were to use OPNsense and HAProxy setup. That would mean no need for DDNS either. That would be much simpler then. I would still need to own a public domain name though right for LE to verify the DNS Challenge?
You could do DDNS to make your VPN connection simpler, but it’s not needed for this use case.
You’d need to own the domain, yes. That way you can do the DNS challenge as you said. Using ACME is nice because it handles the DNS challenges through your DNS provider’s API, so it’s automatic. Combining ACME and HAproxy on OPNsense / PFsense also lets you automatically update the LE cert HAproxy uses. You can do this manually of course if you use a separate system, but having them integrated means when the cert renews every ~90 days ACME will copy the new cert over to HAproxy and restart the HAproxy service to apply the new cert, no manual intervention needed.
It’s much more of a set it and forget it type of deployment.
I tried my hand at HAProxy setup but failed miserably. There’s a lot of configuration to be done and not much explanation on the OPNsense documentation about it. Just some screenshots and a few sentences. Even the current UI has a few different tab names and things compared to the screenshots. I tried to apply Tom’s HAProxy+pfSense video settings over to OPNsense, but there’s probably something that I am not doing right.
I guess I will have to learn about HAProxy in detail, before I can do this now – or maybe try my hand at the NGINX plugin for OPNsense. I installed that as well and that too had a lot of configuration needed.
Everything you need pretty much will live in settings under Services → HAProxy
Real Servers are where you setup the IP and port of the service you want proxied, something like this, make sure to check SSL if it’s in use and uncheck Verify SSL Certificate if it doesn’t have a valid cert.
Next is Backend Pools, which has a load of stuff you can pretty much ignore. Under Servers you want to add the Real Server you added previously. You can disable Health Checking if you want. Ignore everything else.
Then you want Rules & Checks → Rules. This is where you’ll match up your condition (URL) to a backend. The only changes you should need to make here are Select conditions and Use backend pool
Last step is to go to Virtual Services → Public Services. These settings pertain to the web server that will be proxying. Primary fields here are the Name (whatever you want), Listen Addresses (IP of the firewall internally), Certificates (your LE cert), Default Certificate, and Rules (the rules this Public Server will serve).
Thank you @gsrfan01 for the detailed response. I truly appreciate it. I had tried similar, but when I Apply or click Test Syntax, it keeps saying “HAProxy configtest found critical errors”. Apparently starting the service then copies the staging conf file to the actual and then haproxy started up. I was just thinking that something is wrong when the Test Syntax kept failing.
However, I cannot connect to my managed switch web console or my bitwarden server. I can access my FreshTomato AP, but the certificate is not from LE, but a self-signed FreshTomato certificate. So I don’t think the HAProxy is working correctly. Once these 2 or 3 start working, I was planning on adding the rest of the services in HAProxy.
Will every Real Server have its own personal Backend Pool? And will every Backend Pool have it’s own Public Service? The assumption here is that I am not doing any load balancing as I just have 1 server for each-- nextcloud, bitwarden, Managed Switch Web Console etc.
One thing I keep reading about HAProxy setup (forum threads + tutorials) is that you need to change the port at which the OPNsense or pfSense WebUI is running from 443 to something else. Is that required in order to set up the whole reverse-proxy with HAProxy?
Sorry I missed this part of it, yes, you’d want to move the HTTPS port for OPNsense off of 443. This is because the server can’t bind the NGINX service for its web GUI and HAproxy on 443. Move OPNsense over to another port like 8443, then retry HAproxy, that’s probably why it’s complaining.
How is DNS setup for these? You’d want to create a DNS A record for say bitwarden.mydomain.com to point to your OPNsense firewall.
Every real server will have it’s own backend pool. The idea being that a single backend pool could load balance multiple real servers. Because this is simple it’s a 1:1 ratio of real servers to backend pools. As for Public Services, I have 2. 1 is bound to my OPNsense’s internal IP, say 192.168.1.1:443, and I’ve added all my rules to that. The second is bound to my OPNsense’s public ip, specifically one of my DDNS records, say home.mydomain.com:443. This only has my Unifi rule added to it, that way I can hit my Unifi SDN controller from the internet. There’s some additional proxying that goes on with this up in Cloudflare as well.
I have created Host Overrides under Services–>Unbound DNS–>Overrides for bitwarden and all other services and made them all point to the OPNsense internal IP
So if the same Public Service is handling all the rules (for the various services), then which certificate are you using and which Default Certificate? a wildcard?
I currently have a separate LE certificate generated for every service. Maybe I should move over to a wildcard certificate too.
Thanks for sticking with me… I am slowly but surely understanding the intricacies of this.
I followed the exact steps that you listed. After Apply, I restarted the HAProxy service. However, when I try to access the services, I just get “Connection Timed Out”. I tested the 3 Real Servers – and I can correctly access the services when I use the IP:port numbers that I have listed for those Real Servers.
I am trying to re-use the same host names that I used earlier with caddy-- not sure if that is the problem. So for my switch, I created a new hostname + dns entry + cert. But trying to access that with switch.myfancydomain.com just forwards me to the OPNsense login page.
Yep I’m using my LE wildcard cert for this, that way it works for all my subdomains and I don’t have to generate one for each service and maintain those at renewal.
Do a ping on the hostname, does it resolve to the IP address you have set in your Public Service’s Listening Address?
Can you confirm you moved off port 443 in System → Settings → Administration → TCP Port
Ok. I might have to look into getting a wildcard cert instead of individual certs for every service too.
pinging the hostnames results in the correct IP address being shown for every service. I am using the same IPs in my Real Server configuration
Yes, I have moved off of 443 as the TCP Port. I had not disabled the HTTP Redirect. I disabled it now, restarted Unbound and HAProxy but still get timeouts when trying to access the services as servicename.myfancydomain.com
I’ll look into setting up wildcard cert and see if that changes anything. What do you use for the following settings in the Public Service that handles all your internal services?
If you ping say bitwarden.mydomain.com it should resolve the the same IP you have your public service bound do, which should be one of the IPs your OPNsense has. In my case I use the IP for my LAN interface.
Ok. My DNS is probably mis-configured which is what may be causing this. And the very reason why I created this thread to clarify my misconceptions about DNS.
After buying my domain name – I went in and changed the domain names on every device on my network to match my external domain name that I purchased. Not sure if that was the correct thing to do.
Now when I ping just the hostname :
[~]── - ping bitwarden
PING bitwarden.mydomain.com (192.168.1.1) 56(84) bytes of data.
From be5970.rcr21.ord04.atlas.cogentco.com (38.104.106.25) icmp_seq=2 Destination Net Unreachable
From be5970.rcr21.ord04.atlas.cogentco.com (38.104.106.25) icmp_seq=24 Destination Net Unreachable
So it seems to be pointing to my OPNsense… but still tries to reach some other external IP. That could also be because I have setup the same A record in my Cloudflare account – because I was using those external A records for Caddy2 reverse proxy. I’ll delete those and try again.
EDIT :
Weirdly, if I try multiple times to ping bitwarden, I get different results:
The way I handle this is that I have an internal domain, based on Active Directory, that I use for all my internal stuff. Say devdom.lcl. All of my hosts internally have that domain.
The only place I use my fancydomain.com domain is DDNS (to update an A record for my Unifi instance), the wildcard cert from LE, and DNS entries pointing to my firewall.
From your pings it sounds like you have the host with the hostname bitwarden, and a DNS entry with Bitwarden. You can find hosts without a DNS server as well as with, assuming they’re on the same layer 2 network (essentially on the same subnet, more or less. Technically a broadcast domain IIRC). So what’s probably happening is that when you’re asking for the IP of that host via DNS sometimes the host replies and lets you know its IP address and others your OPNsense replies with the record you added.
Using a separate, internal only domain will solve this. You can use any TLD, .lcl is pretty common practice these days. There’s nothing stopping you from using a .com or something else you don’t own, but it’ll cause issues if you need to go to that domain in the future, so it’s best to just use a .lcl.
Thanks. I am in the process of changing the internal domain name to use a sub-domain on my external domain. I’ll make sure to not use the same sub-domain anywhere else.
I have changed the domain name on my internal devices, but the pings hostnames still don’t seem to connect to the correct IP. Will need to flush all the DNS cache, but restarting Unbound multiple times hasn’t helped yet…
