ZDNS - scan the entire internet

The absence of a free Reverse DNS (rDNS) lookup database has hindered large-scale DNS research. To overcome this, we've utilised ZDNS, an open-source, high-performance DNS toolkit developed by Stanford University, to create our own rDNS database. To overcome issues like UDP timeouts in rDNS operations, we've devised a technique of randomising the IP space, significantly improving the efficiency of our scanning process.

Leveraging ZDNS for rDNS Lookups Across the Internet

Understanding rDNS it is helpful for various aspects of internet operations and research. Active DNS measurement plays a crucial role in this understanding, enabling us to delve into a providers advertised use of their IPs. One of the components of this ecosystem is Reverse DNS (rDNS), which serves an important role in IP database categorisation and ASN (Autonomous System Number) classification. However, performing rDNS of the entire internet is not a trivial task.

Previously, Rapid7 provided a free database for rDNS lookups, but it has discontinued the offering. This situation has prompted the need to create our own database, calling for for a robust, efficient, and scalable tool to accomplish this task. That's where ZDNS comes in.

Introducing ZDNS

ZDNS, a part of the ZMap.io project, is a potent tool developed by Stanford University to promote the scalability and reproducibility of DNS research. ZDNS is an open-source DNS measurement framework specifically optimised for large-scale research studies of DNS on the public internet. It is capable of resolving 50 million domains in just 10 minutes and querying the PTR records of the complete public IPv4 address space in approximately 12 hours.

This high-performance toolkit offers a modular interface, enabling researchers to safely implement new functionalities. Its architecture is designed to expose DNS lookup chains by performing its recursive resolution. ZDNS supports a command-line interface, facilitating easy interaction and outputting results in JSON, a machine parsable format.

Enhancements by ZDNS

ZDNS's architecture and feature set are tailored to meet the challenges of extensive DNS research. Its guiding principles ensure that the DNS lookup chain is exposed, and the tool is safe, easy to use, and extensible.

ZDNS's performance optimisations make it a suitable tool for DNS experiments that require querying a large number of names. Parallelism, UDP socket reuse, and selective caching are some of the critical performance optimisations that enable ZDNS to efficiently handle large volumes of DNS queries.

ZDNS's scalability, execution time, and success rate have been evaluated against several existing tools, showcasing its superior performance. For instance, when it comes to exposing the DNS lookup chain, ZDNS is 85 times faster than Dig. ZDNS also outperforms other higher-performance tools, achieving 2.6 to 3.6 times more successful queries per second and experiencing about 30% less packet drop than MassDNS.

Our rDNS Journey

When we started scanning the whole internet using a process called rDNS, we hit a big roadblock - our scans were really slow because of something called UDP timeouts. Our system was waiting around too long for responses from parts of the internet that were either empty or broken.

We came up with two smart solutions. Firstly, instead of scanning the internet's addresses in order, we mixed them up and scanned randomly. This spread out our requests and stopped the system from getting stuck on troublesome parts. Secondly, we checked smaller sections of the internet first, so we didn't waste time waiting for big chunks of the internet that weren't responding.

With these changes, we managed to scan the whole internet in just 13 days, finding over a billion addresses. This goes to show, even when faced with a huge task, a bit of clever thinking can make it possible.

Wrapping Up

ZDNS has proven to be an invaluable tool for DNS research, especially for substantial tasks like performing a reverse DNS scan of the entire internet. Our experience underscores the necessity of inventive solutions when dealing with large-scale challenges, like randomising the IP space to avoid the delays caused by UDP timeouts.

Being an open-source tool, ZDNS is readily available on Github for anyone to use. We highly recommend those interested to further explore its functionality by reading the award-winning paper presented at IMC 2022.

Our work with ZDNS not only demonstrates its potential in DNS research but also illuminates the intricacies of large-scale DNS operations. By adopting a strategic approach, we were successful in mitigating timeout issues and enhancing the efficiency of our scanning process, underscoring the effectiveness of ZDNS.