Dan Kaminsky is understandably swamped today, given the unexpected early release of information about the critical DNS flaw he discovered that potentially affects the security of every website on the internet.
But he found some time to speak with Threat Level about how he discovered the vulnerability that has system administrators scrambling to patch before an exploit -- which is expected to go public by the end of today -- is widely available.
Kaminsky discovered the bug by chance about six months ago, which he promptly disclosed to people in the DNS community. At the end of March, an emergency summit was convened at Microsoft's headquarters, gathering 16 people from around the world to discuss how to address the problem.
On July 8, Kaminsky held a press conference announcing a multi-vendor patch and urging DNS server owners to upgrade their software with the patch immediately. But he declined to disclose details of the bug until next month, when he plans to deliver a talk about the flaw at the Black Hat security conference. Until then, Kaminsky asked researchers not to speculate about the bug, to avoid giving hackers information that could help them exploit it.
Thirteen days after that press conference, however, the security firm Matasano inadvertently released details about the bug on a blog post that the company quickly removed, but has been re-posted elsewhere.
I spoke with Kaminsky about that disclosure, among other issues.
Threat Level: So how pissed off are you?
Dan Kaminsky: (Laughs) I am not the important part here. The important thing is that people patch.
I have to be blunt. The drama is fun and interesting and cool, but it's a distraction. (The important thing is that) it's a really bad bug that really impacts every website you use and your readers use. It impacts whether or not readers are even going to see the article you're about to write. Now I could get into a big fight with lots of people ... and that might happen at some point! But it's a distraction from right now, which is, you know, we did good. We got 13 days of a patch being out without the bug being public. That's unprecedented. I'm pretty proud of at least 13 days. I would have liked 30, but I got 13 ... But the circumstances of how it went public are not what's important today. There will be a time for that, just not now. What is important now is people need to patch.
TL: There were a lot of people who balked at patching because they didn't know the details of the bug.
DK: Well you know, there were people who said, 'Dan, I wish I could patch but I don't know the bug and I can't get the resources I need to patch it.' Well you know the bug now.
You know, Verizon Business has a blog entry where they say that the greatest short-term risk from patching DNS was from the patch itself, from changing such a core and essential element to their systems. I know this. I was a network engineer before I was a security engineer. So that's why we took such extraordinary lengths to try to get people as much time as possible (to patch their systems). There's just a lot of complexity in doing something on this scale. This is something I think a lot of people don’t realize. It was difficult to get the patches even written, let alone get them all released on a single day.
But let me tell you, the complete lack of whining from the (DNS software) vendors ... if I could have gotten as little whining from the security (professionals) ... no I'm not going to say that. It's so tempting! I'm simply going to say this in positive terms. I wish everybody could be as cooperative and understanding and as helpful as Microsoft and ISC (the Internet Systems Consortium) and Cisco and everyone else was who worked so hard to get customers what they needed to protect our networks.
TL: How did you come across the bug? You said in the press conference on July 8 that you hadn't even been looking for this. So what were you doing when you found the bug?
DK: If you look at the history of my talks ... one year I had done some stuff on triangular routing. It's where you have multiple hosts that are all trying to host the same data and you want the fastest one to host it.
So I'm working at this, and I'm wondering if I can, like, use DNS races to figure out the fastest name servers to provide data. I started thinking about this trick I had done (before) with CNAMES -- they're an alias in DNS.
I realized I could look up a random name, and then whichever random name won would override the record for www.mywebsite.com. Essentially, I was looking for a faster way to host data on the internet and I remembered I have ways of overwriting which record the name server uses for 'www' by looking up something else and having it overwrite. And then I thought about that for a second. Wait, it's going to overwrite whatever is www.mywebsite.com! This kind of has security implications! Because if it works you can get around all of our DNS cache-poisoning protections. Then it worked!
I first tried it about six months ago. It took a couple of days to get working. I wrote it in Python to begin with and it was pretty slow. Then I rewrote it in C and it wasn’t slow anymore. It was a couple of seconds. That's when I realized I had a problem.+
TL: Then what did you do?
DK: I looked at it for a while, talked to a couple of really, really trusted people about it. Eventually I went to Paul Vixie (of ISC).
I've been ... looking at other issues with DNS for some time and I had already been working with Vixie on some of the fallout from last year's talk, when I was talking about DNS re-binding attacks. So I go to Paul and I say, Listen, we've got a bigger problem. And I send him the code and the packets and the details. And then there's that moment of, Yeah, we do have a problem.
Paul's an institution in the DNS realm and he basically goes ahead and contacts everybody and brings in Florian Weimer from Germany and brings in representatives from Cisco, Open DNS ... And we start talking on (an e-mail) thread for a couple of weeks about what the implications of this are. A couple of weeks in we realized we should probably have a summit and we should probably have it soon. So I asked Microsoft if they'd provide hosting and they absolutely agreed. On February 20 I had mailed Paul Vixie. And on March 31, 16 people from around the world were in Microsoft headquarters.
When I say there was no b.s. from the vendors, there was just no b.s. from the vendors. They got it. They understood they were in trouble. We skipped past the entire "Is it really a bug?" phase, that's still continuing in public (discussions).
TL: But you’ve got to understand why people said that. You acknowledged that in not disclosing the details, you opened yourself up to people being skeptical about the bug.
DK: People are allowed to be very, very skeptical. But, you know, don't be so skeptical that you're telling people to not patch.
This is a really bad bug. And for everyone who (says), Oh, I knew about this years ago . . . no, you didn't. Stop pretending you did. Because every time you say it, another network doesn't patch (their system).
This (attack takes) ten seconds to hijack the net. . . . Unless you like other people reading your e-mail, go patch. If you want to actually see Google and Yahoo and MySpace and Facebook and the entire web, if you actually want to see the correct web sites, go patch. The debate about whether this bug is new or old is ultimately useless. In ten seconds, the ISP DNS servers are taken over.
TL: It was kind of pie-in-the-sky to think that everyone was going to sit on their hands for 30 days and not post information about what they thought the bug was wasn't it?
DK: You know, a lot of people did. The guys who were actually smart enough to find the bug (didn't disclose it). The people who have been complaining have been people who couldn’t figure it out.
The people who could figure it out e-mailed me privately. And that says a lot. . . . The people who were good enough to figure out the bug by themselves I am incredibly gracious and appreciative of them for mailing me and helping me get the thirteen days that I got.
TL: How quickly did you get the first response from someone who discovered what the bug was?
DK: It was a couple of days.
TL: How far along are people in patching the DNS servers? Do you know how many have been patched?
DK: Way more than I ever would have hoped, (but) less than I would have liked. We were in the high double digits (in terms of percentages). We were getting some pretty good pickup on this patch. The last time I looked at people who were testing against my site it was somewhere in 30 to 40 percent . . . people who were going to my site to test their name servers.
There are a couple million name servers on the internet. There are many million more that are not physically on the internet but are behind firewalls. Ultimately any name server that is not patched is vulnerable and will probably eventually be attacked. The attack is just too good and too easy. My grandma's going to be in the audience (at Black Hat). My grandma's going to understand the bug.
See Also:
