DNS resolution with Wireguard Site to Site VPN using pfSense

So I was previously using a single laptop with wireguard installed to connect to create a wireguard tunnel to a remote pfSense server. With this setup on the laptop it was fairly easy to control DNS resolution within the laptop wireguard instance as I would add the pfSense unbound DNS server to the configuration:

DNS = 10.0.1.1, domain.com

10.0.1.1 represents the unbound local IP address running on pfSense.

Fast forward to a just a tad more involved setup years later where at the second site I installed pfSense which is also running unbound for DNS resolution. So now I have a setup:
—Site #1 - pfsense with unbound server with local DNS host overrides
—Site #2 - pfsense with unbound server with local DNS host overrides

I’ve managed to configure a site-to-site VPN using Wireguard between the two pfsense installations. My problem however is with DNS resolution as I’d ideally like clients on either site to be to DNS resolve by first consulting the local pfsense installation, then if entry isn’t found consult the remote site pfsense, and then finally if not found use some sort of remote external DNS server (9.9.9.9) for name resolution. For some reason I can’t get this setup working.

I’ve searched posts on this forum and came across one post where @LTS_Tom mentioned that Site #1 DNS server should pull and update it’s own DNS records from Site #2 and update the local DNS records accordingly. This sounds great in theory but I can’t figure out how to do this with pfSense.

With each pfSense installation I’ve tried adding the local IP address of the other pfSense to the DNS resolver section within System->General Setup->DNS Server Sections. Within this list however on each site I have external DNS servers additionally listed such as 9.9.9.9 and 1.1.1.1. This setup however doesn’t seem to quite work.

When attempting a dns lookup locally hoping to get the local DNS address from the remote site, I can receive the proper IP address is specifically the DNS server:

nslookup test.domain.com 10.0.1.1

however if not specifying the specific DNS server:

nslookup test.domain.com

the local unbound server will resolve using an external DNS server such as 1.1.1.1 or 9.9.9.9.

Is there an easy way to accomplish what I want to do? In theory this seems like it should be easy if I could specify an order of preference of DNS lookups such as local DNS server, wireguard remote DNS, remoted DNS servers – or – have site 1 pull DNS records from site 2 and update it’s own DNS records accordingly – however I can’t seem to find a way to do this within pfSense.

Option 1: Just remove the external DNS backup servers (1.1.1.1 etc) from DHCP. You don’t need three layers of DNS. Promise.

Option 2: I don’t think you can do master/slave configs in pfsense. Someone will correct me if I am wrong. To do this you would need to setup Bind in a container or VM. I would only do this if you have a rather elaborate internal zone file(s). Or better yet you want to learn. That is always a good reason.

Only issue with removing external DNS servers – both pfsense boxes updated their WAN ip address to cloudflare via dynamic dns updater. The wireguard connections for the S2S connection use DNS names that need an external DNS to provide the WAN IP address. It’s kind of like a chicken and the egg argument here right?

I don’t know a lot about BIND, definitely a project however would this be a potential solution to my issue?

Sounds like you are highlighting the temporary delay updating DNS. This is going to happen with any DNS resolver, local or external. So there is no harm in removing external DNS (odds are your local DNS is just a forwarder anyway). You don’t want your end users jumping between the three DNS resolvers you have selected in DHCP for each query. That is not how this works. Just give your clients your internal DNS server(s) and let that resolver do the lookups.

Bind will let you do anything that is possible with DNS, at the cost of making you learn it. For your replication question, I would go (have gone) with a hidden master setup. Where you have one slave server at each site handling all the grunt work, each providing backup for the other. Nobody talks to the master except the two slaves. This can scale out as far as you want, with any configuration you want.