Takedown

One fine night in November 2011 I got an opportunity to get my hands dirty, working on a project for the United States Federal Bureau of Investigation (FBI). They were planning to seize a bunch of computing assets in New York City that were being used as part of a criminal empire that we called “DNS Changer” since that was the name of the software this gang used to infect a half million or so computers. I work for Internet Systems Consortium (ISC), a small non-profit company headquartered in California. ISC is best known for our work on the Domain Name System (DNS) and our DNS software (called BIND), but we have a growing Internet security practice as well. My task that night in New York City was to install two replacement DNS servers supplied and operated by ISC. This was important because the victims of DNS Changer were dependent on the assets that the FBI needed for evidence, and none of us wanted a half a million DNS Changer victims to “go dark.” It was a little odd for ISC to send me—ISC’s Chairman and Founder—on this job, but rank hath its privileges.

It was a very long night, since there was no way to complete a detailed plan before the takedown began. After the DNS Changer gang was in custody and I could “go intrusive” on their equipment, it took me a couple of hours to figure out exactly how everything was wired together and to move the first group of victims over to ISC’s replacement DNS servers. It then took a couple more hours to move and test the rest of the victims. All this long night I had a cell phone headset in one ear and a half dozen chat windows open on my laptop—the full takedown team was worldwide and there were other actions occurring elsewhere. By the time we were done and it was safe to power off the DNS Changer equipment, it was 7am and I nearly missed my train. Note to self, if another chance comes along to run—huffing and puffing—through the New York City subway system and Penn Station, trying to keep up with a younger and better conditioned member of FBI’s New York division—take it! But maybe next time bring better shoes.

Cleanup

Since the original court order that authorized ISC to install and operate these replacement DNS servers was due to expire on March 9 2012, a new DNS Changer Working Group (DCWG) was formed to handle victim notification and remediation. We had roughly four months to identify and notify half million or so DNS Changer victims, and to help these victims clean up their infected computers. Many victims would have to reinstall Windows on their computers—which at first was the only sure cure for this particular infection. On top of that, many of the victims have had their DSL or Cable modems (“home routers”) reconfigured by the DNS Changer malware, so that they were using ISC’s replacement DNS servers even if none of their computers are still infected and even if none of their computers were running Windows. Most Internet users do not have the skills necessary to check and repair the configuration of their home routers, and most Windows users are also unwilling to reinstall Windows. So, even when we could identify and notify a victim, we had a hard time “closing the deal”.

We didn’t make it. When March 9 2012 loomed, we still had hundreds of thousands of victims dependent on ISC’s replacement DNS servers. Therefore the FBI asked the judge for an extension and we were given four more months. No fooling around this time, there won’t be another extension, it’s now or never, put up or shut up, etc. Noting that no private company or individual can legally operate this replacement DNS service on the open Internet unless they have a judge’s permission to do so, many ISP’s are now starting up replacement DNS servers inside their own networks, accessible only by their own customers, in order to control the risks they would otherwise face on July 9 2012 when the second and final court order is due to expire. But that kind of risk management isn’t the same as cleaning up the problem. I don’t think we want to “kick this can down the road”. If an ISP wants to run a replacement DNS server for the purpose of forcibly breaking these computers, in small batches, to get their owners to call in and ask for help, that’s one thing. But if it’s just going to be a new permanent service that the ISP offers to these customers, count me as “opposed.”

We as a digital society are much better at strategies for coping than we are at strategies for remediation.

Is your DNS OK?

A half dozen national Internet security teams around the world have created special web sites that will display a warning message to potential victims of the DNS Changer infection. For example if you visit http://dns-ok.de/ then you’ll get a German language page saying either that you appear to be infected or that you appear not to be infected. Andrew Fried and I created http://dns-ok.us/ for the same purpose, though of course our page is in American English. The full list of these “DNS Checking” web sites is published on the DCWG’s web site along with a lot of information about the threat, the arrests, the takedown, the court orders, and clean-up information for victims. Now that we’ve got all these web sites that are able to tell someone if they are a victim and that tell victims what to do to clean up their computers and their home routers, the problem seems to be getting people to care.

Internet users are endlessly bombarded with warnings about their security and with offers of services and software (some of it apparently “free”) offering to make their computers healthier. The victims of DNS Changer are by this time jaded or overwhelmed or both. The Internet seems to be a very dangerous place, and most Internet users probably feel that they could spend more than half their waking hours just installing patches and responding to warnings—unless they just put their heads down, ignore all that noise, and try instead to get their work (or play) done. I am sympathetic to this mindset. The problem is, the Internet really is that dangerous, and people really do need to pay more attention to the dangers of unpatched or infected computers. Given that most people can’t take the time to care enough about these dangers, their infected computers become a threat to everybody else, thus completing the cycle of dangerousness begetting more dangerousness.

All those within the sound of my voice, please check out the DCWG web site and find out if your DNS is OK. Ask your customers, your friends, and your family to do likewise. Or use this as an excuse to go visit the people in your life less technical than yourself, and show them how to check their DNS.

July 9 and Beyond

On July 9 2012 the replacement DNS servers operated by ISC will be shut down and any victims who still depend on these servers will face new risks. Notice I’m not saying that they “will go dark” since that’s not entirely clear. Some of them will go dark, some of them will face long delays on every web page they visit, some might not show any symptoms at all. The long term risk I foresee is that some new criminal empire (or more than one) will offer services to replace ISC’s, and they will easily recapture a large part of the DNS Changer victim population. There are ways to do this that don’t leave tracks—so not every criminal who does this will be automatically and immediately detected, arrested, and charged. I would like to see these computers cleaned up, so that they don’t pose a lasting but latent threat to the rest of us.

Speaking of lasting, latent threats to the rest of us, I was part of the Conficker cabal recently immortalized by Mark Bowden’s book, “Worm.” We still don’t know the identities of any of the criminals who foisted Conficker on an unready world back in 2008. But we do know that the victim population has not dropped below six million (6,000,000). So we still collect the “sinkhole” data about these victims, we still report on it to network operators, and every year we buy another rack of disk drives to hold the next year or so worth of data. We’re out of ideas for how to get people to care that their computers are infected with Conficker. These victims seem to feel that have more important things to worry about. My gut feeling is that they’re wrong, but I can’t seem to prove it. My other gut feeling about all this is that we, as a digital society, are doing this all wrong.