Hi guys, tried to get help on Netbird’s Github, but no replies have been forthcoming yet, so I thought I’d try my luck here
I’m self-hosting Netbird on my home server with Authentik, running in an Ubuntu Server VM (internal IP 192.168.7.200), behind an nginx reverse proxy. I’m also running Adguard Home as my default resolver on my main router, advertising it via DHCP to all LAN clients (192.168.7.1 on port 53).
All is running fine - I can get to my Netbird dashboard, add new nodes etc.
However - if I advertise the above DNS to my Netbird Network, I’m getting the the following lines in client container:
2024-12-30T21:56:20Z WARN [error: read udp 192.168.7.200:56413->192.168.7.1:53: i/o timeout, upstream: 192.168.7.1:53] client/internal/dns/upstream.go:116: got an error while connecting to upstream
2024-12-30T21:56:20Z ERRO client/internal/dns/upstream.go:149: all queries to the upstream nameservers failed with timeout
If I set the DNS to for example Quad9, all is working, so it only happens with my internal DNS resolver. Meanwhile, nslookup inside the container CAN use 192.168.7.1 to resolve domain names. The same timeout issue occurs if run a netbird client directly on the VM. Nslookup on the VM also has no trouble using the internal DNS resolver. Plus, I spun up Headscale, and Tailscale clients have no issue using my internal DNS resolver.
So… I’m wondering, what I can do… I want to use my internal resolver so that I can take advantage of DoT and ad filtering. Thanks!
# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 192.168.7.1
nameserver 192.168.7.1
And as I said, doing an nslookup to any domain works through 192.168.7.1:53
Since adguard is seeing the traffic we can assuem that the responses from adguard to the specific dns client to not get routed properly. I’d check 2 things on the adguard container host:
(1) are the routes properly set so that the responses will take the right direction? (Docker might be overriding the route to the client using its own interface routes)
(2) if the netbird container and the adguard container are on the same host, and you are using some host local firewall like UFW, you might has a firewall issue. Try turning off that host local firewall for a moment and check if it then works.
Adguard isn’t running in a container on the same host as the Netbird client - it’s running directly on my main OpenWRT router on port 53 as the system-wide DNS resolver (advertised by DHCP). It directs Internet-bound queries to the Quad9 and Cloudflare DoT resolvers, and local-bound queries to dnsmasq also on the router on port 54. This setup has been working for years for all clients - Wireguard, other machines on my LAN, Tailscale clients (one of which is running as a container on the same host as the Netbird client container, with the same network settings for the container).
I’ve checked UFW is not active on the VM running the Netbird client. And, what’s strange is that running nslookup works in the client container’s shell (docker exec).
What I’d now do is look at the DNS traffic between the netbird container and the adguard machine, if that gives you any ideas? Maybe adguard does not respond at all, or the response is malformed…
use the Wireshark ssh feature to record the traffic on the openwrt machine interface where the netbird machine can be reached.
Is it safe to share a partial export (wireshark pcapng file) of a Wireshark capture here? I’m seeing some suspicious packets that could be the answer, but I’m not knowledgeable enough to properly parse what is happening…
EDIT: solved!!
Found out the problem… I had in the DNS settings in Adguard Home this: [//]192.168.7.1:54
An empty domain specification, // has the special meaning of “unqualified names only”, i.e. names without any dots in them, like myhost or router. Those will only be used for resolving requests for unqualified domain names, but not their subdomains.
And based on a hunch after thinking and seeing Netbird’s requests, I commented this out and only left in [/local/]192.168.7.1:54 and [/lan/]192.168.7.1:54, and voilá Netbirds now finally works with my Adguard Home.
Thanks @xerxes for the Wireshark tip, it saved me after days and days of failing
