DNS Malware Filtering Compared: Quad9 VS Cloudflare VS DNS Filter VS OpenDNS / Cisco Umbrella

Domains list was from here:

https://isc.sans.edu/suspicious_domains.html

Pastebin CSV results file here

Bash Script Used

#!/bin/bash
# Bulk DNS Lookup
# Generates a CSV of DNS lookups from a list of domains.
#
# File name/path of domain list:
domain_list='domains.txt' # One FQDN per line in file.
#
# IP address of the nameserver used for lookups:
ns1_ip='1.1.1.1' # Cloudflare
ns2_ip='9.9.9.9' # Quad9
ns3_ip='1.1.1.2' # Cloudflare Malware
ns4_ip='103.247.36.36' # DNS Filter
ns5_ip='208.67.222.222' # OpenDNS / Cisco Umbrella
#
# Seconds to wait between lookups:
loop_wait='1' # Is set to 1 second.

echo "Domain name, $ns1_ip,$ns2_ip,$ns3_ip,$ns4_ip,$ns5_ip "; # Start CSV
for domain in `cat $domain_list` # Start looping through domains
do
    ip1=`dig @$ns1_ip +short $domain |tail -n1`; # IP address lookup DNS Server1
    ip2=`dig @$ns2_ip +short $domain |tail -n1`; # IP address lookup DNS server2
    ip3=`dig @$ns3_ip +short $domain |tail -n1`; # IP address lookup DNS server3
    ip4=`dig @$ns4_ip +short $domain |tail -n1`; # IP address lookup DNS server4
    ip5=`dig @$ns5_ip +short $domain |tail -n1`; # IP address lookup DNS server5
            echo -en "$domain,$ip1,$ip2,$ip3,$ip4,$ip5\n";
#    sleep $loop_wait # Pause before the next lookup to avoid flooding NS
done;
4 Likes

Very interesting video. Would it not be useful to also measure average speed results?

2 Likes

Iā€™m shocked at the OpenDNS performance. Guess I need to switch to Quad 9.

3 Likes

I tried switching my PFSense to use Quad9 as the primary DNS but for some reason it isnā€™t grabbing it. Itā€™s defaulting to my ISP anyone have a suggestion?

1 Like

I was very surprised to see these results as well. This is exactly the reason to do testing. Excellent video. Iā€™ve never come across Quad 9 before, but Iā€™ll definitely be looking into them now. Thanks!

4 Likes

That was a very good video, I enjoyed it a lot.

2 Likes

Not to be a conspiracy theorist ā€“ but isnā€™t there some connection between Quad9 and the NSA or other government (usually 3 letter) agencies?

1 Like

Hi! Just saw the video. Thanks for the review! Iā€™m the Executive Director of Quad9. Iā€™ve got a few answers and comments based on the video.

First: Weā€™re a very small 501( c )3 non-profit, but we punch well above our weight in a lot of ways despite having only a fraction the capitalized size of any of the other services you put in the list. Weā€™re always looking for sponsors, both individual and corporate - see our webpage for the big friendly ā€œdonateā€ button.

Second: Weā€™re a really great security service, but we also arenā€™t selling personal data, nor do we have some hidden plan by giving away our service for free. Weā€™re not upselling you to a paid plan, nor are we promoting our other hosting or DNS or marketing, nor are we selling or digesting your private data. In fact, we never even transmit your IP address out of the cluster of systems in each city where we receive it, nor do we ever write it to any sort of storage. Furthermore, we have no accounts, no signup, and no way of tracking who are users are - itā€™s entirely free, and we really donā€™t even want to know who are users are. The downside of that is that there isnā€™t any customization or reporting, because that would imply tracking information and storage. Weā€™re explicitly designed not to be able to retain personal data from the ground up. This is actually a very appealing part of our project for schools, public networks, and of course everyone in the areas covered by GDPR regulations, and I think weā€™re the only service in the world operated by a non-profit with these goals.
To the person who references rumors that we are part of a vast government conspiracy: No. Our staff, and board, and sponsors are all working towards making Quad9 the most private service available. Specifically, everyone working here would depart if anything nefarious like that were the case, and Iā€™m sorry that our reputations for privacy fanaticism canā€™t easily be represented in a clickable format. We find that some of our goals may align with certain law enforcement organizations who focus on financial crimes (stopping cyber-crime is fundamental to both of our missions) but thatā€™s the only place where the Venn diagrams cross. We are currently funded only by private donations, mostly by cyber-security companies who value our work, individuals, and quite a bit of in-kind donations from other non-profits and service providers for network capacity.

Third: Our security coverage is pretty great, and Iā€™m pleased to see your numbers bear that out. The reason weā€™re so awesome is because of our threat intelligence partners. We donā€™t actually evaluate threats and figure out who is doing what - we ingest lists from around 19 different companies, and some open lists and then our job is to relay/apply that data via our DNS servers. This is very different than everyone else, who may be building the lists themselves, or who have only a limited number of providers. The companies that give us these lists of malicious/phishing/command&control/etc. hosts is because we give them some insights as to the volumes of ā€œhitsā€ happening on the domains they give us. We donā€™t tell them anything about your personal data (no IP addresses) but they find the volume data to be super-useful with our many millions of end users giving immediate validation to growing or falling trends in malicious domains. Also, you can find out who provided us with any domain that is blocked by going to our web page - thereā€™s a search bar right on the front. We donā€™t have any ā€œhiddenā€ suppliers of threat data - everything is public about individual domains, though we donā€™t publish the list itself.
I suspect the SANS list is being ingested by one or more of our TI providers and analyzed for false positives, and then re-bundled and sent to us in part. We donā€™t (yet) take that list directly. This is actually quite common - finding false positives is a science all to itself, and we are extremely sensitive to FP entries. Many of the public lists are more prone to FP data, so we constantly are looking for issues in the data and working with both open and closed list providers to keep false positives low or nonexistent.

Fourth: Weā€™re in more than 150 cities and 90 countries now, and still growing, so thereā€™s probably a server close to you, and you get routed there automatically. Anycast is fantastic.

Fifth: We support DNS-over-TLS, DNS-over-HTTPS, and DNSCrypt protocols for extra interception-proofing. The newest Chrome being rolled out & Android =>Pie will auto-upgrade to encrypt DNS connections to Quad9, and the very early beta versions of Windows as well. Encryption is good!

Sixth: We have multiple ā€œflavorsā€ of the service. There is 9.9.9.10, which provides ā€œvanillaā€ DNS with no blocklist similar to CF and Googleā€™s service, and in fact there is even no DNSSEC on that address to make it even more able to resolve everything, even things it shouldnā€™t (user beware!) Thereā€™s also 9.9.9.11, which has blocking + ECS, which is a whole different story but the summary is that some CDNs will work better to get you to the closest server at the cost of some privacy leakage (the ECS protocol sends some parts of your IP address to the remote DNS server during the lookup.) Also, we try to encourage everyone to set up their ā€œsecondaryā€ server as well as the backups to minimize effects of routing issues or maintenance windows.

9.9.9.9
149.112.112.112
2620:fe::fe

Thanks again for the review - Iā€™ll try to keep my eye on this thread to answer any questions.

16 Likes

No. But once a rumor like that gets started, itā€™s hard to kill. But no, absolutely not. For SO many reasons. Weā€™re privacy fanatics.

7 Likes

Iā€™m in Thailand. Latency on 9.9.9.9 is 188ms whereas latency on 1.1.1.1 is 32 ms. On your website it indicates where you do not have a server, the DNS request will be forwarded to your nearest server. Geographically Cambodia seems to be that, but Singapore is closest latency wise. What is the basis of calculating nearest server and should things improve?

2 Likes

Thank you very much @quad9dns for taking the time to post in the forums and also thank you for such a great service!

8 Likes

There were quite a few YouTube comments asking about NextDNS. I setup an account and then I ran the query against their services. While they did better than DNS Filter, Umbrella, and Cloudflare Filtered, Quad9 is still by far the winner in this test.

4 Likes

@Deku In the pfSense menu under System / General Setup, clear the checkbox ā€˜Allow DNS server list to be overridden by DHCP/PPP on WANā€™ in the ā€˜DNS Server Settings sectionā€™

1 Like

Donā€™t need to wait to switch over to QUAD9 methinks

1 Like

Put a screenshot up of your System / General Setup.

1 Like

Thank you for taking time to address the issue. Itā€™s nice to here information from the company directly.

2 Likes

LTS_Tom Yeomanā€™s work on this thanks. Used OpenDNS but switched when they were acquired by Cisco, primary 9999 with the secondary set to 1111 now as of moments ago all Quad9. Was surprised at the results of OpenDNS seems Cisco isnā€™t putting much except marketing into Umbrella.
You got some attention with this video including Quad9. The response from them was the kind of response we in the biz like.
SANS has some great free data sources but their course offerings are a bit expensive.

2 Likes

I had been using Quad9, but switched to 1.1.1.2 because of the lower ping times from Cloudflare, but after this video Iā€™m switching back. A slight speed difference isnā€™t enough to give up effective filtering.

2 Likes

Most DNS providers do have two IPā€™s for their service so you can use them as primary and secondary. Quad9ā€™s other IP is 149.112.112.112

2 Likes

Really interesting, thanks for the comparison.

I went ahead and ran the script against pfBlockerNG. Slightly better than Quad9, but still let a few through.

pfBlockerNG

2 Likes