Understanding DNS & DDNS

gsrfan01 · March 28, 2021, 8:29pm

Try an ipconfig /flushdns on the hosts. They’ll cache DNS results do it doesn’t have to query the server for something it’s resolved already.

inxsible · March 29, 2021, 5:26pm

gsrfan01:

The way I handle this is that I have an internal domain, based on Active Directory, that I use for all my internal stuff. Say devdom.lcl. All of my hosts internally have that domain.

The only place I use my fancydomain.com domain is DDNS (to update an A record for my Unifi instance), the wildcard cert from LE, and DNS entries pointing to my firewall.

From your pings it sounds like you have the host with the hostname bitwarden, and a DNS entry with Bitwarden. You can find hosts without a DNS server as well as with, assuming they’re on the same layer 2 network (essentially on the same subnet, more or less. Technically a broadcast domain IIRC). So what’s probably happening is that when you’re asking for the IP of that host via DNS sometimes the host replies and lets you know its IP address and others your OPNsense replies with the record you added.

Using a separate, internal only domain will solve this. You can use any TLD, .lcl is pretty common practice these days. There’s nothing stopping you from using a .com or something else you don’t own, but it’ll cause issues if you need to go to that domain in the future, so it’s best to just use a .lcl.

I have now disabled the DDNS service temporarily until I can fix the DNS mess.

I am using *.fancydomain.com in LE – to generate the wildcard cert.
I have DNS entries for all my hosts under Unbound DNS → Overrides → Host Overrides as below:

The IP points to my internal OPNsense IP and the domain is fancydomain.com

Also, you are correct, that I have a bitwarden host (LXC container under proxmox) and I also was using the same name to be able to access the service. Should I give it different names – Is that what the problem is? I have now changed all the domain names on every device to be lan.fancydomain.com

I still get timeouts on every service I try to proxy via the HAProxy.

I don’t have any windows devices. I have flushed the cache in the Archlinux boxes. I also tried creating a brand new profile in firefox just in case the browser had cached the DNS entries, but still no luck.

inxsible · March 29, 2021, 5:34pm

Here are the ping results for bitwarden :

[~]── - ping -c2 bitwarden
PING bitwarden.lan.fancydomain.com (192.168.1.25) 56(84) bytes of data.
64 bytes from bitwarden.lan.fancydomain.com (192.168.1.25): icmp_seq=1 ttl=64 time=0.597 ms
64 bytes from bitwarden.lan.fancydomain.com (192.168.1.25): icmp_seq=2 ttl=64 time=0.272 ms

--- bitwarden.lan.fancydomain.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.272/0.434/0.597/0.162 ms

[~]── - ping -c2 bitwarden.fancydomain.com
PING bitwarden.fancydomain.com (192.168.1.1) 56(84) bytes of data.

--- bitwarden.fancydomain.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1014ms

[~]── - ping -c2 bitwarden.lan.fancydomain.com
PING bitwarden.lan.fancydomain.com (192.168.1.25) 56(84) bytes of data.
64 bytes from bitwarden.lan.fancydomain.com (192.168.1.25): icmp_seq=1 ttl=64 time=0.253 ms
64 bytes from bitwarden.lan.fancydomain.com (192.168.1.25): icmp_seq=2 ttl=64 time=0.298 ms

--- bitwarden.lan.fancydomain.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1020ms
rtt min/avg/max/mdev = 0.253/0.275/0.298/0.022 ms
[~]── -

I just get 100% packet loss whenever I try to ping anything on the opnsense internal IP.

[~]── - ping -c2 opnsense
PING opnsense.lan.fancydomain.com (192.168.1.1) 56(84) bytes of data.

--- opnsense.lan.fancydomain.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1005ms

[~]── - ping -c2 opnsense.fancydomain.com
PING opnsense.fancydomain.com (192.168.1.1) 56(84) bytes of data.

--- opnsense.fancydomain.com ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1019ms

Kadarin · March 29, 2021, 7:46pm

Thank you for that post. First time I’ve seen HAProxy interface, and it looks extremely straightforward. I’m far too used to NetScalers. (Anyone needs help with those, ping me.)

gsrfan01 · March 30, 2021, 11:24am

Your OPNsense is probably denying ICMP echo requests so it won’t reply back, that’s pretty normal for firewalls. Mine replies, but you might have a rule for that. You can turn on logging for your rules and go to Firewall → Log File → Live View and see it.

ERR_CONNECTION_TIMED_OUT or are you getting a 503 Service Unavailable error? If you’re getting the CONNECTION_TIMED_OUT that’s normally that you’re trying to go to a port that OPNsense isn’t listening on, 503 would be from HAproxy. Your Public Server listening address should look like this:
Assming you’re in the 192.168.1.1 range.

inxsible · March 30, 2021, 6:02pm

I don’t have any rules to explicitly block ICMP – I just have 2 allow rules as below apart from the auto-generated rules:

Yes. I do get the ERR_CONNECTION_TIMED_OUT.
My public server listening address is set up correctly as you have listed it. Should I also add 192.168.1.1:80 in there too?

gsrfan01 · March 30, 2021, 7:45pm

I ran through the HAproxy setup from after changing the HTTPS port to 10443 quick in a lab so everything can be seen in full. You should only need to do the below before the video picks up:

Install HAproxy
Change OPNsense from 443 to 10443, disable HTTP redirect
Setup ACME for Let’s Encrypt

inxsible · March 30, 2021, 10:05pm

Thanks for creating that video but I have set up HAProxy in exactly the same way. Here are the screenshots for the bitwarden service, but I have set up similar for apnet (my AP), switch(managed switch web console), guac(Apache guacamole), ups and home(heimdall home page container)

Real Server:

Backend Pool 1st

Backend Pool 2nd

Condition

Rule

Public Service 1st

Public Service 2nd

Public Service 3rd

Only thing I changed is the actual domain to fancydomain in the Condition (for the purposes of the screenshot). I still only get ERR_CONNECTION_TIMED_OUT when accessing any of these EXCEPT switch.

Accessing switch.fancydomain.com redirects me to switch.fancydomain.com:10443 and shows me the opnsense login page albeit with the WRONG certificate (shows me cert for opnsense.fancydomain.com)

EDIT: BTW, I have deleted all the A records from my external DNS provider (Cloudflare). I do have the DNS A record entries under Unbound for each of the services that I want HAProxy to serve.

gsrfan01 · March 30, 2021, 11:04pm

That sounds fishy, is 10443 the port used for the switch? Do you have any other plugins on OPNsense?

Out of curiosity, if you run an NMAP intensive scan on your firewall, under port 443 is it showing your LE cert? You should see ssl-cert: Subject: commonName=.fancydomain.com*

inxsible · March 30, 2021, 11:34pm

10443 is the new port I moved the Opnsense WebUI on since it could no longer be on 443 due to HAProxy setup. I did this via System–>Settings–>Administration–>TCP Port. I also disabled the HTTP Redirect checkbox under the same heading as you mentioned.

I do, I have the Let’s Encrypt plugin(os-acme-client), the DDNS(os-dydns, which was installed by default), UPS (os-nut) & SMART (os-smart)

Here’s the nmap result:

[~]── - sudo nmap -p 443 -T4 -A -v -Pn 192.168.1.1
Starting Nmap 7.91 ( https://nmap.org ) at 2021-03-30 18:24 CDT
NSE: Loaded 153 scripts for scanning.
NSE: Script Pre-scanning.
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Initiating ARP Ping Scan at 18:24
Scanning 192.168.1.1 [1 port]
Completed ARP Ping Scan at 18:24, 0.04s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 18:24
Completed Parallel DNS resolution of 1 host. at 18:24, 0.00s elapsed
Initiating SYN Stealth Scan at 18:24
Scanning nextcloud.fancydomain.com (192.168.1.1) [1 port]
Completed SYN Stealth Scan at 18:24, 0.23s elapsed (1 total ports)
Initiating Service scan at 18:24
Initiating OS detection (try #1) against nextcloud.fancydomain.com (192.168.1.1)
Retrying OS detection (try #2) against nextcloud.fancydomain.com (192.168.1.1)
NSE: Script scanning 192.168.1.1.
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Nmap scan report for nextcloud.fancydomain.com (192.168.1.1)
Host is up (0.00023s latency).

PORT    STATE    SERVICE VERSION
443/tcp filtered https
MAC Address: A0:36:9F:D7:95:69 (Intel Corporate)
Too many fingerprints match this host to give specific OS details
Network Distance: 1 hop

TRACEROUTE
HOP RTT     ADDRESS
1   0.23 ms nextcloud.fancydomain.com (192.168.1.1)

NSE: Script Post-scanning.
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Initiating NSE at 18:24
Completed NSE at 18:24, 0.00s elapsed
Read data files from: /usr/bin/../share/nmap
OS and Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 3.49 seconds
           Raw packets sent: 51 (6.788KB) | Rcvd: 4 (316B)

So, it’s returning the detection against nextcloud.fancydomain instead of opnsense.fancydomain even though nextcloud is not located at x.x.1.1 but rather opnsense is. Each time I run the above nmap command it returns 1 of the services running on my network. It’s not nextcloud each time.

gsrfan01 · March 31, 2021, 12:05pm

That’s the way to do it, but you shouldn’t be getting redirected to 10443 like that. Your best bet might be to disable / remove all of the servers from HAproxy except 1 and get that working as intended.

Do you have NextCloud.yourfancydomain.com pointing to the firewall? Running the same command on Windows but swapping to my IP I don’t get it resolving to anything. That might be some Windows idiosyncrasy though.

inxsible · March 31, 2021, 6:07pm

I do, under the Unbound DNS -->Overrides → Host Overrides. Eventually once the apnet, switch etc start working, I do plan to move all the services to be proxied via HAProxy which is why I have those entries.

EDIT: If I change the Unbound DNS overrides to point to the actual IP of the said service, I can correctly reach the webpage for that service – which is what I expected. So it seems that HAProxy is not correctly sending the request to the backend service due to some mis-configuration on my part.

I have listed all the screenshots of my HAProxy configuration… Not really sure which port HAProxy is listening on but I set it at the 192.168.1.1:443. I have tried rebooting the firewall and restarting Unbound and HAProxy multiple times by now.

EDIT2: I have now even tried the HAProxy ListenAddress to

127.0.0.1:443
127.0.0.1:80
0.0.0.0:443
0.0.0.0:80
192.168.1.1:443
192.168.1.1:80

I have even tried a few different combinations of the above but may not have hit every permutation and combination. Still the same with the ERR_CONNECTION_TIMED_OUT.

I saw a new OPNsense update released today – one of the plugins that got upgraded was haproxy so I thought I might as well upgrade to the latest and try again. But still no dice !!!

gsrfan01 · March 31, 2021, 10:43pm

That makes sense why it’s swapping around between your services I suppose since that IP would have multiple DNS entries.

I’m curious if your firewall logs might show something under Firewall → Log Files → Live View filter by dst contains [firewall ip] and dstport contains 443

inxsible · April 1, 2021, 8:16pm

NordVPN		Apr 1 15:02:53	10.8.2.3:8084	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 15:02:53	10.8.2.3:44597	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:14:02	10.8.2.3:3097	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:12:38	10.8.2.3:11106	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:11:49	10.8.2.3:56339	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:11:49	10.8.2.3:26731	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:11:34	10.8.2.3:18799	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:10:44	10.8.2.3:46538	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:10:44	10.8.2.3:19906	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:05:10	10.8.2.3:38315	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:05:10	10.8.2.3:8054	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)	
NordVPN		Apr 1 14:03:53	10.8.2.3:45927	192.168.1.1:443	tcp	let out anything from firewall host itself (force gw)

I don’t see any incoming to the firewall… I only see outgoing requests. Each time I refresh the page for any of my services – bitwarden.fancydomain.com for example – I see a few entries in the logs, but none are incoming.

My NordVPN interface is the VPN client that I use. I also use the NordVPN interface as the Outgoing interface for DNS queries to authoritative servers.

I see this in my HAProxy logs:

2021-04-01T23:23:56	haproxy[31670]	192.168.1.1:15343 [01/Apr/2021:23:23:56.605] https~ https/<NOSRV> -1/-1/-1/-1/0 503 222 - - SC-- 1/1/0/0/0 0/0 "GET /ocs/v1.php/cloud/user HTTP/1.1"	 
2021-04-01T01:00:00	haproxy[31670]	192.168.1.1:51433 [01/Apr/2021:01:00:00.361] https~ https/<NOSRV> -1/-1/-1/-1/0 503 222 - - SC-- 1/1/0/0/0 0/0 "GET /ocs/v1.php/cloud/user HTTP/1.1"

How do I read that? Is there an error?

inxsible · April 2, 2021, 4:11am

telnet results in

[~]── - telnet 192.168.1.1 443
Trying 192.168.1.1...
telnet: Unable to connect to remote host: Connection timed out
[~]── - 
[~]── - telnet 192.168.1.1 80
Trying 192.168.1.1...
telnet: Unable to connect to remote host: Connection timed out
[~]── -

So nothing seems to be listening on port 443 or 80.

I might have to have a look at the gateways or firewall rules but afaik, I am not blocking anything on the LAN network.

gsrfan01 · April 2, 2021, 4:29pm

If I had to hazard a guess, Nord might be messing with things, or at least making it more complicated.

You should see 2 requests, 1 from your client to the firewall and 1 from the firewall to the server

I’m using these filter options

Can you test without going through Nord, or is all traffic routed through it? You may have to have HAproxy listen on the NordVPN interface IP.

inxsible · April 3, 2021, 1:09am

I do have all my traffic routed through NordVPN except my work computer which goes out via the ISP but is also on a completely different network and IP range.

I’ll try and shut down the Nord interface and see if I can route everything out of the ISP. Switching from caddy2 as reverse proxy to HAProxy turned into a major headache for me, I’d say. But I am learning too… so there’s that. Thanks for your consistent help @gsrfan01

How would I do that? I didn’t see any setting under HAProxy that would allow me to change the interface. Should I just use the NordVPN tunnel network IP in the ListenAddress for HAProxy?

inxsible · April 3, 2021, 6:00pm

@gsrfan01 you were correct about the VPN client causing issues. When I switched the gateway on the LAN Rules to use the default Gateway, I was able to get to the services and was also able to see the wildcard cert that I created for my domain name. So the above works.

However, if I change the Gateway in the LAN rule to anything but default — I keep getting timeouts, even when using the WAN_DHCP as the gateway. Obviously I cannot keep it at default, because then it would negate the use of a VPN client.

Is there a way to use the VPN gateway but stil be able to get the HAProxy working ?

gsrfan01 · April 3, 2021, 11:09pm

You could possibly create a rule to route traffic to that IP to use the default gateway and put it above your VPN rule.

I don’t route everything, just specific clients, but here is my rule:

Basically I have an alias for all the IPs I want out my WG VPN WG_VPN_Hosts and an alias of RFC1918 (internal / reserved IP scopes). When the WG hosts hit that rule it checks if they’re bound for that RFC1918 list of networks. If they are, it ignores it and traffic continues out to the default gateway. If it is, it sends it out my WG gateway and tags the traffic. There’s a floating rule that denies any tagged traffic to any gateway that’s not the WG VPN in the event it drops.

Probably the easiest way to do that, but the above would be proper more “proper”.

inxsible · April 4, 2021, 3:42am

Thanks @gsrfan01

I added a rule where if the source is LAN net, and destination is RFC1918, it would route it via the default gateway.

It works – however I am not 100% sure why the default gateway would change the behavior like that.

This thread started out as a DNS/DDNS clarification and evolved into HAProxy setup and ended up being an issue with using policy based routing…

Thanks for sticking by me…