Skip to content

Understanding DNS Requests on macOS

Tags: development, learning, software • Categories: Software

Table of Contents

The DNS Stack

Here’s a quick overview of how the DNS stack on macOS works. Much of this also applies to linux.

  • cat /etc/resolv.conf contains a list of potential DNS resolvers
  • DNS servers work off of UDP (not TCP) on port 53, but at this point, there are multiple DNS protocols that use different ports and configurations.
  • You can see the DNS resolvers assigned to a specific device using scutil --dns
  • This list is autogen’d and managed by the DNS settings of the current network device
    • Unlike linux variants, this is not managed by NetworkManager and instead uses macOS-specific tooling, much of which is opaque.
    • networksetup -getdnsservers Ethernet to get manually configured DNS servers, does not return DNS servers set by DHCP
  • Unless you override the DNS in the network settings, you’ll use the DNS servers handed out by your DHCP lease (this is configured on your router, which is generally pulled upstream from your ISP).
  • A "search domain" is automatically appended to an unqualified domain name (i.e. no TLD). These search domains can be handed out via a DHCP lease.
    • This is how tailscale’s magic DNS works. Tailscale overwrites your resolv.conf and adds a search domain. When you access yourbox it appends it’s tailnet domain so becomes yourbox.the-name.ts.net

DNS CLI Tools

Some built-in tools:

  • ping is a great way to check mDNS resolution
  • nslookup apple.com
  • dig apple.com
  • host apple.com
  • dscacheutil -q host -a name apple.com you’ll notice that this one is not like the others. This will become very important later on (this is a macOS-native DNS command).

Most of these tools are somewhat old and harder to use than they need to be. Here are some more modern tools that are helpful to install, especially if you want to inspect DoH, DNSEC, DoT, and other related acronyms.

Here are some examples of how to use these to run various DNS tests:

  • To test a specific DNS server response use dig @192.168.7.100 google.com where 192... is the IP address of the DNS server in question
  • dnslookup example.org https://1.1.1.1/dns-query to test DoH on a DNS server
  • drill example.org -D to validate DNSSEC response for the host
  • Not a CLI tool, but https://one.one.one.one/help/ is great for testing what protocol your DNS setup is using.
  • dns-sd -Q on.quad9.net TXT will determine if your DNS resolver stack is using Quad9
  • If you are using Quad9, dns-sd -Q proto.on.quad9.net TXT determines what protocol you are using.
    • Really need use of dynamic responses to DNS requests!

Isolating The Problem

In my case, my problem was very similar to this post:

  • Multiple DNS resolvers, the first of which was a local IP address running pihole
  • I noticed that pihole (sometimes!) was being ignored dispite it being the first entry in the resolver list.
  • dig domain.hole returned the correct result but http domain.hole failed while http 192.168.7.10 'Host: domain.hole' worked, which was very confusing. dig was not returning the value used by the operating system.
  • I noticed that sometimes Chrome would properly load the local domain which meant it was using my local DNS server. I believe this is because macOS seems to always prefer DNSSEC if the upstream server supports it and there does not seem to be a way to disable it.
  • I disabled all of the local relay and tracking privacy stuff.
  • host and nslookup returned the same value as dig but http and any browser-based interactions failed.
  • However, sudo dns-sd -Q domain.hole returned 0.0.0.0 and dscacheutil -q host -a name domain.hole returned an empty response. Boom! this is the core of the problem.

This is what caused me to go down this deep dark rabbit hole of attempting to understand what was happening with macOS DNS.

It looks as if macOS DNS resolution is not used when running dig, host, nslookup and friends. dscacheutil runs the "macOS native" DNS lookup. mDNSResponder is supposed to me

What I discovered (detailed below) was:

  1. The order of DNS servers specified in the network device is not respected
  2. Instead, if there are multiple servers, and if one of them supports a more secure protocol, that is used instead

What this means is if you are running a local pi-hole installation, which does not support a more secure protocol (like DoH or DoT), it will be ignored by your macOS devices unless it’s the only DNS server used as a resolver.

The problem with using pi-hole as your only resolver is if it goes down for some reason nothing on the network will be able to connect. I’ll leave setting up a local DoH server for another post.

Digging into the Details of macOS DNS

Viewing Private Log Data

macOS has centralized all logging behind a new mechanism that you can access via the log cli tool. However, private data—including important low-level DNS information—is hidden. You used to be able to easily turn this on:

sudo log config --mode "private_data:on"

This doesn’t work anymore.

You need to install a profile that enables a Enable-Private-Data config. Note that even after you download + open the profile, you need to manually enable the profile.

Debug mDNSResponder Using System Logs

Now that we can see private data, let’s enable debug logging:

sudo log config --subsystem com.apple.mDNSResponder --mode "level:debug"

Now you can view more verbose debug logs:

log stream --predicate 'subsystem == "com.apple.mDNSResponder"' --debug

This seems to ‘hard reset’ the mDNS system in a way that spews out a bunch of configuration logs:

 sudo killall -9 mDNSResponder mDNSResponderHelper

Here’s what my logs look like:

2024-09-18 07:20:02.462842-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] Updated DNS services (8)
2024-09-18 07:20:02.463029-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (1/8) -- id: 12, type: Do53, source: sc, scope: none, interface: /0, servers: {100.84.131.117:53, 192.168.7.32:53, 9.9.9.9:53}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, use count: 1
2024-09-18 07:20:02.463070-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (2/8) -- id: 15, type: DoH, source: dns, scope: none, interface: /0, servers: {}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, resolver config: {provider name: 9.9.9.9, provider path: /dns-query}, connection hostname: dns.quad9.net, port: 443, parent: 12
2024-09-18 07:20:02.463112-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (3/8) -- id: 16, type: DoT, source: dns, scope: none, interface: /0, servers: {}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, resolver config: {provider name: 9.9.9.9, provider path: }, connection hostname: dns.quad9.net, port: 853, parent: 12
2024-09-18 07:20:02.463140-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (4/8) -- id: 13, type: Do53, source: sc, scope: interface, interface: en7/19, servers: {100.84.131.117:53, 192.168.7.32:53, 9.9.9.9:53}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, use count: 1
2024-09-18 07:20:02.463166-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (5/8) -- id: 20, type: Do53, source: sc, scope: interface, interface: en0/15, servers: {100.84.131.117:53, 192.168.7.32:53, 9.9.9.9:53}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, use count: 1
2024-09-18 07:20:02.463193-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (6/8) -- id: 10, type: ODoH, source: nw, scope: uuid (45EA5412-B365-4F5A-8919-1B83B3611010), interface: /0, servers: {}, domains: {}, attributes: {a-ok, aaaa-ok, connection-problems, fail-fast, allows-failover}, interface properties: {ipv4}, resolver config: {provider name: mask.icloud.com, provider path: /dns-query}, use count: 1
2024-09-18 07:20:02.463218-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (7/8) -- id: 11, type: ODoH, source: nw, scope: uuid (68B564AC-DA55-4EC9-B40E-528AFAC32561), interface: /0, servers: {}, domains: {}, attributes: {a-ok, aaaa-ok, connection-problems, fail-fast, allows-failover}, interface properties: {ipv4}, resolver config: {provider name: mask-boot.icloud.com, provider path: /dns-query}, use count: 1
2024-09-18 07:20:02.463244-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (8/8) -- id: 17, type: ODoH, source: nw, scope: uuid (0AF6E7D2-87EF-4C38-B5A1-003B264686F5), interface: /0, servers: {}, domains: {}, attributes: {a-ok, aaaa-ok, connection-problems, fail-fast, allows-failover}, interface properties: {ipv4}, resolver config: {provider name: mask-boot.icloud.com, provider path: /dns-query}, use count: 1
2

Note that without the privacy profile listed above installed, you won’t see this! It’s all <private> in the logs without this installed.

This reload operation also seems to take place when you set DNS servers (but only if you change the configuration):

sudo networksetup -setdnsservers "Ethernet" 192.168.7.32 9.9.9.9

What’s interesting here is:

  • Do53 is your standard UDP port 53 DNS server
  • DoT is DNS over TLS and is a competing standard with DoH and seems worse (easier to network sniff) which is why you don’t see it much.
  • DoH is mentioned which means macOS is inspecting the DNS servers for DoH support
  • ODoH is new to me "oblivious" DNS over HTTP. The primary improvement seems to be that the requestor IP is never exposed to the resolver.
    • This is great if you are running a scraping operation. Bot blockers will use fancy techniques (like randomized subdomains) to expose your originating IP to detect scraping behavior.
    • In my configuration, ODoH was not chosen as the resolver. More info on that below.
  • DNS servers are assigned IDs that are used downstream in the logs. This is why you never see a server IP address in the logs.
    • This makes sense given that DoH, ODoH, etc all point to the same server but use a separate protocol so a DNS server IP is not an effective UID.
  • I noticed some logs mentioning tracking. Some of the macOS built-in tracking protection seems to operate at this level of the system.
  • A "successful" response (i.e. a real IP) and an unsuccessful response both look identical from a logging perspective.

If you watch the logs you’ll see entries like:

2024-09-18 07:24:02.159227-0600 0x6c8cf5d  Default     0x0                  60448  0    mDNSResponder: [com.apple.mDNSResponder:Default] [R10533->Q62016] Question for <mask.hash: 'U6iZVNFQ4PSzZhzDy1JxVg=='> (HTTPS) assigned DNS service 15

This indicates that the DoH service which is not my pi hole. Interesting.

If I set DNS servers to CloudFlare (which supports both DoH and DoT)

sudo networksetup -setdnsservers "Ethernet" 1.1.1.1

Then only three DNS resolves are generated and notably, DoH is not generated, which seems to indicate that macOS prefers DoT:

2024-09-18 07:41:10.515974-0600 0x6ce2953  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] Updated DNS services (3)
2024-09-18 07:41:10.516023-0600 0x6ce2953  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (1/3) -- id: 1, type: Do53, source: sc, scope: none, interface: /0, servers: {1.1.1.1:53}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, use count: 1
2024-09-18 07:41:10.516048-0600 0x6ce2953  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (2/3) -- id: 3, type: DoT, source: dns, scope: none, interface: /0, servers: {}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, resolver config: {provider name: 1.1.1.1, provider path: }, connection hostname: one.one.one.one, port: 853, parent: 1
2024-09-18 07:41:10.516086-0600 0x6ce2953  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (3/3) -- id: 2, type: Do53, source: sc, scope: interface, interface: en7/19, servers: {1.1.1.1:53}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, use count: 1
2

If I use Quad9 servers (which support DoT and DoH as well):

sudo networksetup -setdnsservers "Ethernet" 9.9.9.9

DoH is chosen. Very odd! There must be something on the quad9 side hinting to use DoH instead of DoT:

2024-09-18 07:44:54.274414-0600 0x6ce2af1  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] Updated DNS services (5)
2024-09-18 07:44:54.274444-0600 0x6ce2af1  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (1/5) -- id: 6, type: Do53, source: sc, scope: none, interface: /0, servers: {9.9.9.9:53}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, use count: 1
2024-09-18 07:44:54.274466-0600 0x6ce2af1  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (2/5) -- id: 8, type: DoT, source: dns, scope: none, interface: /0, servers: {}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, resolver config: {provider name: 9.9.9.9, provider path: }, connection hostname: dns.quad9.net, port: 853, parent: 6
2024-09-18 07:44:54.274536-0600 0x6ce2af1  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (3/5) -- id: 9, type: DoH, source: dns, scope: none, interface: /0, servers: {}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, resolver config: {provider name: 9.9.9.9, provider path: /dns-query}, connection hostname: dns.quad9.net, port: 443, parent: 6
2024-09-18 07:44:54.274570-0600 0x6ce2af1  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (4/5) -- id: 7, type: Do53, source: sc, scope: interface, interface: en7/19, servers: {9.9.9.9:53}, domains: {.}, attributes: {a-ok}, interface properties: {ipv4}, use count: 1
2024-09-18 07:44:54.274617-0600 0x6ce2af1  Default     0x0                  40692  0    mDNSResponder: [com.apple.mDNSResponder:Default] DNS service (5/5) -- id: 5, type: ODoH, source: nw, scope: uuid (45EA5412-B365-4F5A-8919-1B83B3611010), interface: /0, servers: {}, domains: {}, attributes: {a-ok, aaaa-ok, fail-fast, allows-failover}, interface properties: {ipv4}, resolver config: {provider name: mask.icloud.com, provider path: /dns-query}, use count: 1
2

macOS Tracking and ODoH

Strangely, ODoH is not well supported in the open-source ecosystem. Many command line DNS utilities (dig, dog, nslookup, etc) do not support it. The "official" CloudFlare client has been archived.
However, q does support it although I could not get an example request working

Based on the logs above, it looks like Apple runs their own proxy service for ODoH requests:

resolver config: {provider name: mask.icloud.com, provider path: /dns-query},

When ODoH is working, you will see the proxy that is used for each request:

2024-09-18 08:22:39.346773-0600 0x6d3f307  Default     0x0                  33953  0    mDNSResponder: [com.apple.mDNSResponder:Default] [R1759->Q19429] Question for <mask.hash: 'AkvVROlKp9zC+SOnL953ow=='> (Addr) assigned DNS service -- id: 5, type: ODoH, source: nw, scope: uuid (353A4F9C-CEF5-4CEE-93AD-4697BB0318D7), interface: /0, servers: {}, domains: {}, attributes: {a-ok, aaaa-ok, fail-fast, allows-failover}, interface properties: {ipv4}, resolver config: {provider name: odoh.cloudflare-dns.com, provider path: /dns-query}, use count: 1

It seems like apple sometimes uses CloudFlare’s ODoH proxy by default.

In order to return on ODoH you need to enable it in three locations:

  1. iCloud settings
  2. Network settings
  3. If you are using Safari, their a specific setting for Safari.

Here’s the icloud settings page:

What’s interesting is when I toggled iCloud relay and privacy tracking on my ethernet device (in network settings) the ODoH configuration was not generated in the mDNSResponder logs. Something funky is going on there.

I found this list of public ODoH servers useful for testing.

This document written by Apple was interesting and mentions setting up a local DNS override for mask.icloud.com if you want ODoH on your local network.

Clearing DNS Cache

  • chrome://net-internals/#dns to (manually) clear DNS cache
  • It does not look like macOS caches DNS information at all. If you change dns servers (i.e. sudo networksetup -setdnsservers "Ethernet" 192.168.7.100 it seems to instantly pull the changed records). I did not test if this is true when you change networks.
  • Here’s how to definitely destroy all DNS services on macOS `dscacheutil -flushcache; sudo killall -HUP mDNSResponder; sudo killall -9 mDNSResponder mDNSResponderHelper
  • In Safari, you may have to "empty caches" from the UI in order to ensure DNS cache is competely cleared.

Open Questions

  1. How does macOS interrogate an IP and determine what protocols it supports? What process does it follow? This process was confusing to me and I’d be curious to understand how it works.
  2. Does linux use a similar process of ignoring resolving order and picking the most secure protocol instead?
  3. Does DoH protect DNS requests from geographical lookup IP exposure? This is important if you do any sort of scraping, especially from a residential IP as a proxy assigned to a Chrome profile will not cache DNS requests by the operating system.
  4. How can you set up a local DoH server with SSL support which uses a local pi-hole DNS server as its upstream DNS server?

Keep in Touch

Subscribe to my email list to keep in touch. I’ll send you new blog posts and other thoughts.