Google domains vs Cloudflare DNS with ndots:5 in resolv.conf

Hey everyone, I had quite the head scratcher for the past couple weeks which I’ve been able to narrow down to being caused by Cloudflare DNS. For context, I have my home network setup with the same domain name as my external domain, we’ll call it my-domain.net using pfSense. And I’m trying to get K3s up and running, but for some reason I’m not privy to, kubernetes uses ndots:5 in the resolv.conf file which is causing my domain to get appended to all queries. E.g. running a nslookup on google.com would result in google.com.my-domain.net when I’m setup using the Cloudflare DNS, but would resolve correctly on google domains (as far as I can tell, they have the same records setup). I wanted to use Cloudflare for some of the nice features like proxying and actually supporting an API that I can use with Lets Encrypt, but this became a bit of a show stopper for me.

Anyone have any ideas why the Google DNS would resolve the way I’d expect, and why Cloudflare doesn’t? As far as I can tell, Google actually responds with a NXDOMAIN when I query a subdomain that doesn’t exist while Cloudflare “seems” to respond as if it does. Here’s some debug output I have:

Google Domains

❯ dig +ndots=5 +search google.com

; <<>> DiG 9.16.24 <<>> +ndots +search google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25281
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             102     IN      A       142.250.72.78

;; Query time: 20 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Feb 05 14:13:40 MST 2022
;; MSG SIZE  rcvd: 55


❯ nslookup -debug google.com
Server:         10.0.0.1
Address:        10.0.0.1#53

------------
    QUESTIONS:
        google.com.my-domain.net, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  my-domain.net
        origin = ns-cloud-d1.googledomains.com
        mail addr = cloud-dns-hostmaster.google.com
        serial = 11
        refresh = 21600
        retry = 3600
        expire = 259200
        minimum = 300
        ttl = 64
    ADDITIONAL RECORDS:
------------
** server can't find google.com.my-domain.net: NXDOMAIN
Server:         10.0.0.1
Address:        10.0.0.1#53

------------
    QUESTIONS:
        google.com, type = A, class = IN
    ANSWERS:
    ->  google.com
        internet address = 142.250.190.110
        ttl = 212
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:   google.com
Address: 142.250.190.110
------------
    QUESTIONS:
        google.com, type = AAAA, class = IN
    ANSWERS:
    ->  google.com
        has AAAA address 2607:f8b0:4009:80b::200e
        ttl = 133
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   google.com
Address: 2607:f8b0:4009:80b::200e

Cloudflare

❯ nslookup -debug google.com
Server:         10.0.0.1
Address:        10.0.0.1#53

------------
    QUESTIONS:
        google.com.my-domain.net, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  my-domain.net
        origin = damian.ns.cloudflare.com
        mail addr = dns.cloudflare.com
        serial = 2269578432
        refresh = 10000
        retry = 2400
        expire = 604800
        minimum = 3600
        ttl = 2585
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
------------
    QUESTIONS:
        google.com.my-domain.net, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  my-domain.net
        origin = damian.ns.cloudflare.com
        mail addr = dns.cloudflare.com
        serial = 2269578432
        refresh = 10000
        retry = 2400
        expire = 604800
        minimum = 3600
        ttl = 3405
    ADDITIONAL RECORDS:
------------
*** Can't find google.com.my-domain.net: No answer


❯ dig +ndots=5 +search google.com 

; <<>> DiG 9.16.24 <<>> +ndots +search google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19994
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.my-domain.net.                IN      A

;; AUTHORITY SECTION:
my-domain.net.            1943    IN      SOA     damian.ns.cloudflare.com. dns.cloudflare.com. 2269578432 10000 2400 604800 3600

;; Query time: 3 msec
;; SERVER: 10.0.0.1#53(10.0.0.1)
;; WHEN: Sat Feb 05 14:13:33 MST 2022
;; MSG SIZE  rcvd: 115

Cloudflare not responding with NXDOMAIN is because of DNSSEC on your ext domain - https://blog.cloudflare.com/black-lies/

I dont run kubernetes, but there seems to be a requirement for some images where if an NXDOMAIN is not returned it cannot resolve. https://github.com/kubernetes/kubernetes/issues/64924

Unsure if this is because you have ext and internal domains the same though, possibly. I always set internal domains to subdomain of external to avoid issues eg ad.my-domain.net (if I’m running a windows domain).

The kubernetes link seemed to have a few solutions you could try, but as I dont use kubernetes, I’d read up thoroughly before trying something random.

Good luck

1 Like

Thanks for the feedback! I think I may have a working solution now. I moved my domain over to lan.my-domain.net and added a custom option in the pfSense resolver to keep the lan subdomain internal by replying with NXDOMAIN:

server:
local-zone: "lan.my-domain.net" static

Super weird that this is still an issue 4 years after people started complaining. I can’t imagine I’m the only one to use my external domain as my internal domain and using Kubernetes :person_shrugging:

:+1:

Nice, glad you have a solution. Seems strange, but I don’t know kubenetes well enough to know why it’s set that way.