Why DNS in OS X 10.10 is broken, and what you can do to fix it

Recently, there has been a lot of discussion about the current state of Apple's software quality. Anyone with even a passing familiarity with development knows that bugs are par for the course, and most people aren't bothered by small, day-to-day bugs that are fixed within a reasonable timeframe. Obviously, like everyone else, Apple's software has its share of those.

But there's another category of bug—glaring, perplexing bugs that couldn't possibly have escaped the attention of the software engineers in question, let alone the quality assurance department. Such issues exist, and sometimes they go unfixed for months. Or years. Or ever. Hopefully, the set of network issues with OS X 10.10 described below won't fall into this column, but they do raise an obvious question: why?

For 12 years, the mDNSResponder service managed a surprisingly large part of our Mac's networking, and it managed this task well. But as of OS X 10.10, the mDNSResponder has been replaced with discoveryd, which does the same thing. Mostly. Here are some strange networking problems we've observed since installing 10.10:

Random failures to resolve DNS names. This is rare, but every once in a while, Safari stops loading any and all websites. Turns out that the OS X DNS resolver has stopped working. (Command line tools such as nslookup, host, or dig still work because those use their own DNS lookup code.)

Duplicate machine names. We use an old Mac named "nirrti" as a file- and iTunes server. In the pre-10.10 days, once in a blue moon nirrti would rename herself to "nirrti (2)", presumably because it looked like another machine was already using the name "nirrti". Under 10.10, this now happens a lot, sometimes getting all the way to nirrti (7). Changing back the computer name in the Sharing pane of the System Preferences usually doesn't take. Apart from looking bad, this also makes opening network connections and playing iTunes content harder, as you need to connect to the right version of the name or nothing happens.

Very inconsistent wake on demand. In OS X 10.6, Apple introduced the "wake on demand" feature where a sleeping Mac automatically wakes up when it gets accessed over the network. This is great, because now you can let a computer that functions as a server sleep. Before 10.10, this worked reasonably well. Usually, our Apple TV would play content from the iTunes library on nirrti without trouble. But as of OS X 10.10, wake on demand sometimes works for a while, but most of the time it doesn't.

Impossible to reach services running on a Mac from the outside. When on the road, we like to share the screen of a home Mac to monitor the progress of ongoing downloads and the like. Under 10.10, this wouldn't work at all over IPv4, only over IPv6.

Very flakey Wide Area Bonjour registration. In order to connect to that home machine, we have to register its name and services using Wide Area Bonjour/dynamic DNS updates. Again, pre-10.10 this worked fairly reliably, but under 10.10 it's hit-and-miss, with more "miss" than "hit".

At first blush, you'd assume that these were all separate, smaller problems. However, that's not the case.

mDNSResponder versus discoveryd

Resolving DNS names, resolving Bonjour machine names, resolving Bonjour service advertisement and discovery, and opening ports in NAT gateways to allow incoming network connections are all jobs that have been performed by the mDNSResponder daemon. This was introduced with Mac OS X 10.2 Jaguar back in 2002. mDNSResponder is written in C and has been released as open source by Apple, and it has found its way to all kinds of non-Apple operating systems and hardware.

Again, as of OS X 10.10, mDNSResponder has been replaced by discoveryd. Curiously, discoveryd is (re)written in C++, not exactly one of Apple's favorite languages. It's not on Apple's list of open source projects. And apparently, the responsible people at Apple haven't been reading their Joel on Software. As per Joel Spolsky's dire warnings against rewriting software from scratch, discoveryd has its share of bugs. For instance, when the system is unable to resolve DNS names, discoveryd log messages indicate that it doesn't recognize the replies from the DNS server to its own requests. discoveryd is also a frequent guest in the CrashReporter logs.

One of the functions of the mDNSResponder was to talk to a home gateway and ask it to forward incoming network connections for any services running on the Mac in question. In the non-Apple world, this is done with the uPnP IGD protocol, and Apple had its own NAT-PMP protocol for this. More recently, the IETF has standardized PCP, a new protocol that also works with IPv6 and firewalls, not just IPv4 and Network Address Translation (NAT) as used by home routers. mDNSResponder supported all of them, but it looks like discoveryd doesn't use any of them. As such, if you enable screen sharing on your Mac and try to connect to it from elsewhere on the Internet, your home router doesn't know where to send the incoming request. Thus, the screen remains unshared.

If discoveryd isn't doing its job properly, sometimes it helps to restart it. However, even this doesn't always work, and restarting discoveryd gets old fast. If you're not scared of a little command line hacking, there's a better solution: you can reinstall the OS X 10.9 mDNSResponder on a 10.10 system. And after that, all the problems listed above magically go away.

Restoring mDNSResponder, if you dare

First a word of warning: if something goes wrong, your Mac may not be able to boot. This can be fixed by booting in single user mode and re-enabling discoveryd (see below). Be sure you have a recent backup just in case, and only follow the steps below if you are prepared to boot into single user mode if needed.

The first thing you need to do is find copies of 10.9 mDNSResponder, mDNSResponderHelper, and their plist files. On a 10.9 system, they're found in /usr/sbin/ and /System/Library/LaunchDaemons, respectively. These Terminal commands will copy them to the desktop for easy copying to the 10.10 system on which you want to install them:

cd ~/Desktop/
cp /usr/sbin/mDNSResponder .
cp /usr/sbin/mDNSResponderHelper .
cp /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist .
cp /System/Library/LaunchDaemons/com.apple.mDNSResponderHelper.plist .

Another way to get the mDNSResponder and its helper is from a Time Machine backup. Connect your Time Machine drive or click on the backup image on your (Time Capsule) network drive. Next, navigate to a backup from before October 16, when you were still running OS X 10.9, along these lines:

cd /Volumes/Time Machine Backups/Backups.backupdb/
cd <my machine name>
ls
cd <date/time of backup>
cd Macintosh\ HD
cp usr/sbin/mDNSResponder ~/Desktop/
cp usr/sbin/mDNSResponderHelper ~/Desktop/
cp System/Library/LaunchDaemons/com.apple.mDNSResponder.plist ~/Desktop/
cp System/Library/LaunchDaemons/com.apple.mDNSResponderHelper.plist ~/Desktop/

With these preparations in place, you can copy the mDNSResponder and helper to the OS X 10.10 machine:

sudo cp mDNSResponder /usr/sbin/
sudo cp mDNSResponderHelper /usr/sbin/
sudo cp com.apple.mDNSResponder.plist /System/Library/LaunchDaemons/
sudo cp com.apple.mDNSResponderHelper.plist /System/Library/LaunchDaemons/

Then, deactivate discoveryd and activate mDNSResponder. Remember, sudo makes a command execute with root or system administrator privileges, so it asks for a system administrator password. Don’t worry if you get "Could not find specified service."

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.discoveryd.plist
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.discoveryd_helper.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.mDNSResponderHelper.plist

Now reboot.

Should you wish to reactivate discoveryd, use the following commands. If your machine won't boot properly, boot in single user mode by rebooting while holding command-S and then you'll be able to type these commands. Type "exit" when you're done.

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSResponder.plist
sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.mDNSResponderHelper.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.discoveryd.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.discoveryd_helper.plist

You may be wondering which new OS X 10.10 features no longer work because you're using an older version of such an important system component. As far as we've been able to tell in a short period of testing, all the headline ones all still there—Handoff and AirDrop work in both directions. Enjoy your rediscovered network stability.

Listing image by UWW ResNET on Flickr