Home / Blogs

DNS Changer

Paul Vixie

Takedown

One fine night in November 2011 I got an opportunity to get my hands dirty, working on a project for the United States Federal Bureau of Investigation (FBI). They were planning to seize a bunch of computing assets in New York City that were being used as part of a criminal empire that we called "DNS Changer" since that was the name of the software this gang used to infect a half million or so computers. I work for Internet Systems Consortium (ISC), a small non-profit company headquartered in California. ISC is best known for our work on the Domain Name System (DNS) and our DNS software (called BIND), but we have a growing Internet security practice as well. My task that night in New York City was to install two replacement DNS servers supplied and operated by ISC. This was important because the victims of DNS Changer were dependent on the assets that the FBI needed for evidence, and none of us wanted a half a million DNS Changer victims to "go dark." It was a little odd for ISC to send me — ISC's Chairman and Founder — on this job, but rank hath its privileges.

It was a very long night, since there was no way to complete a detailed plan before the takedown began. After the DNS Changer gang was in custody and I could "go intrusive" on their equipment, it took me a couple of hours to figure out exactly how everything was wired together and to move the first group of victims over to ISC's replacement DNS servers. It then took a couple more hours to move and test the rest of the victims. All this long night I had a cell phone headset in one ear and a half dozen chat windows open on my laptop — the full takedown team was worldwide and there were other actions occurring elsewhere. By the time we were done and it was safe to power off the DNS Changer equipment, it was 7am and I nearly missed my train. Note to self, if another chance comes along to run — huffing and puffing — through the New York City subway system and Penn Station, trying to keep up with a younger and better conditioned member of FBI's New York division — take it! But maybe next time bring better shoes.

Cleanup

Since the original court order that authorized ISC to install and operate these replacement DNS servers was due to expire on March 9 2012, a new DNS Changer Working Group (DCWG) was formed to handle victim notification and remediation. We had roughly four months to identify and notify half million or so DNS Changer victims, and to help these victims clean up their infected computers. Many victims would have to reinstall Windows on their computers — which at first was the only sure cure for this particular infection. On top of that, many of the victims have had their DSL or Cable modems ("home routers") reconfigured by the DNS Changer malware, so that they were using ISC's replacement DNS servers even if none of their computers are still infected and even if none of their computers were running Windows. Most Internet users do not have the skills necessary to check and repair the configuration of their home routers, and most Windows users are also unwilling to reinstall Windows. So, even when we could identify and notify a victim, we had a hard time "closing the deal".

We didn't make it. When March 9 2012 loomed, we still had hundreds of thousands of victims dependent on ISC's replacement DNS servers. Therefore the FBI asked the judge for an extension and we were given four more months. No fooling around this time, there won't be another extension, it's now or never, put up or shut up, etc. Noting that no private company or individual can legally operate this replacement DNS service on the open Internet unless they have a judge's permission to do so, many ISP's are now starting up replacement DNS servers inside their own networks, accessible only by their own customers, in order to control the risks they would otherwise face on July 9 2012 when the second and final court order is due to expire. But that kind of risk management isn't the same as cleaning up the problem. I don't think we want to "kick this can down the road". If an ISP wants to run a replacement DNS server for the purpose of forcibly breaking these computers, in small batches, to get their owners to call in and ask for help, that's one thing. But if it's just going to be a new permanent service that the ISP offers to these customers, count me as "opposed."

We as a digital society are much better at strategies for coping than we are at strategies for remediation.

Is your DNS OK?

A half dozen national Internet security teams around the world have created special web sites that will display a warning message to potential victims of the DNS Changer infection. For example if you visit http://dns-ok.de/ then you'll get a German language page saying either that you appear to be infected or that you appear not to be infected. Andrew Fried and I created http://dns-ok.us/ for the same purpose, though of course our page is in American English. The full list of these "DNS Checking" web sites is published on the DCWG's web site along with a lot of information about the threat, the arrests, the takedown, the court orders, and clean-up information for victims. Now that we've got all these web sites that are able to tell someone if they are a victim and that tell victims what to do to clean up their computers and their home routers, the problem seems to be getting people to care.

Internet users are endlessly bombarded with warnings about their security and with offers of services and software (some of it apparently "free") offering to make their computers healthier. The victims of DNS Changer are by this time jaded or overwhelmed or both. The Internet seems to be a very dangerous place, and most Internet users probably feel that they could spend more than half their waking hours just installing patches and responding to warnings — unless they just put their heads down, ignore all that noise, and try instead to get their work (or play) done. I am sympathetic to this mindset. The problem is, the Internet really is that dangerous, and people really do need to pay more attention to the dangers of unpatched or infected computers. Given that most people can't take the time to care enough about these dangers, their infected computers become a threat to everybody else, thus completing the cycle of dangerousness begetting more dangerousness.

All those within the sound of my voice, please check out the DCWG web site and find out if your DNS is OK. Ask your customers, your friends, and your family to do likewise. Or use this as an excuse to go visit the people in your life less technical than yourself, and show them how to check their DNS.

July 9 and Beyond

On July 9 2012 the replacement DNS servers operated by ISC will be shut down and any victims who still depend on these servers will face new risks. Notice I'm not saying that they "will go dark" since that's not entirely clear. Some of them will go dark, some of them will face long delays on every web page they visit, some might not show any symptoms at all. The long term risk I foresee is that some new criminal empire (or more than one) will offer services to replace ISC's, and they will easily recapture a large part of the DNS Changer victim population. There are ways to do this that don't leave tracks — so not every criminal who does this will be automatically and immediately detected, arrested, and charged. I would like to see these computers cleaned up, so that they don't pose a lasting but latent threat to the rest of us.

Speaking of lasting, latent threats to the rest of us, I was part of the Conficker cabal recently immortalized by Mark Bowden's book, "Worm." We still don't know the identities of any of the criminals who foisted Conficker on an unready world back in 2008. But we do know that the victim population has not dropped below six million (6,000,000). So we still collect the "sinkhole" data about these victims, we still report on it to network operators, and every year we buy another rack of disk drives to hold the next year or so worth of data. We're out of ideas for how to get people to care that their computers are infected with Conficker. These victims seem to feel that have more important things to worry about. My gut feeling is that they're wrong, but I can't seem to prove it. My other gut feeling about all this is that we, as a digital society, are doing this all wrong.

By Paul Vixie, CEO, Farsight Security. More blog posts from Paul Vixie can also be read here.

Related topics: Cybercrime, DNS, Law, Security

WEEKLY WRAP — Get CircleID's Weekly Summary Report by Email:

Comments

"We're out of ideas for how to Frank Bulk  –  Mar 28, 2012 5:00 AM PDT

"We're out of ideas for how to get people to care that their computers are infected with Conficker. These victims seem to feel that have more important things to worry about."

For the same reason, many of those who are knowingly or unknowingly affected by DNSChanger won't do anything until their browsing experience is so bad that they call their ISP or someone to help them. 

As you pointed out, criminals are using this lag time to victimize these users again.  Rather than extend the remediation period, it would have been better to give a week or two to get the word out about these issues to ISPs, and then turn the replacement DNS servers down.

The threat isn't seen as urgent Jason Lewis  –  Mar 28, 2012 8:05 AM PDT

Users are mostly unaware their computers are compromised, so it's not on their radar.

I've attempted to contact larger companies about their Conficker infections.  I can only provide external IP information and timestamps.  Their end users are usually behind proxies and are unable to correlate the info I provide with their internal network.  It seems the accounting on their end is not detailed enough or not available for tracking the compromised hosts.

If their computer is working, why should Phil Howard  –  Mar 28, 2012 10:21 AM PDT

If their computer is working, why should they worry?  After a certain point (and I think that point has been reached, now), you just have to give up on stupid people and let them find their own way.

Phase 2: just return one IP address (of a site that gives information about the issue and how to fix it) for all domains requested.  This site should be phrased to look more like their computer giving them error messages, rather than being redirected to some site like DCWG or FBI.  It should say things like "OOPS, configuration error, invalid DNS server addresses, FIX ME FIX ME".  Add other hints like "Call the IT dept, call the ISP, hire a security consultant".

Phase 3: blackhole

Inertia, it needs to be broken Todd Knarr  –  Mar 29, 2012 1:33 PM PDT

I have to second the sentiments above. Many users judge based simply on whether their computer and Internet access works or not. If it works, to them there just isn't any problem and no need to put time and effort out to fix what to them isn't broken. They won't do anything until their Internet access stops working. So, don't molly-coddle them. Give them a reasonable warning, ideally working with ISPs to get the warning to them through recognizably-official customer-support channels. And when that reasonable lead-time's up, shut it off. The ones that'll respond to warnings and fix things will already have fixed things, and the ones that haven't won't respond to anything less than their Internet going away.

There's a certain point where you just have to say "You knew about it, you ignored it, if you're in trouble now it's your own fault.". If a car owner ignores the mechanic's warning that he needs to have the oil checked and changed regularly, saying "But the car runs fine without that, why should I bother?", then when the car finally breaks down because most of the oil's gone is it the mechanic's fault? Or do we place the blame squarely on the car owner?

I'll "third" that sentiment. There are Frank Bulk  –  Mar 29, 2012 4:38 PM PDT

I'll "third" that sentiment.  There are countless subscribers where we need to perform proactive maintenance on their inside RF cabling that won't return multiple voicemails or respond to a door tag, so we have to suspend their broadband to get their attention.  They're either too busy or figure we got it wrong, so just ignore us instead of communicate with us.  As both Todd and Phil said, communicate a deadline and then stick to it.

http://dns-ok.us/ inaccessible on IPv6 due to peering Kenyon Ralph  –  Apr 06, 2012 10:51 AM PDT

http://dns-ok.us/ is unreachable on IPv6 from my Hurricane Electric location, presumably due to peering problems.

To post comments, please login or create an account.

Related Blogs

Related News

Topics

Industry Updates – Sponsored Posts

New from Verisign Labs - Measuring Privacy Disclosures in URL Query Strings

DotConnectAfrica Delegates Attend the Kenya Internet Governance Forum

3 Questions to Ask Your DNS Host about Lowering DDoS Risks

Continuing to Work in the Public Interest

Verisign Named to the OTA's 2014 Online Trust Honor Roll

4 Minutes Vs. 4 Hours: A Responder Explains Emergency DDoS Mitigation

Dyn Acquires Internet Intelligence Company, Renesys

Tips to Address New FFIEC DDoS Requirements

Smokescreening: Data Theft Makes DDoS More Dangerous

Introducing getdns: a Modern, Extensible, Open Source API for the DNS

Why We Decided to Stop Offering Free Accounts

dotStrategy Selects Neustar's Registry Threat Mitigation Services for .BUZZ Registry

Tony Kirsch Announced As Head of Global Consulting of ARI Registry Services

24 Million Home Routers Expose ISPs to Massive DNS-Based DDoS Attacks

Dyn Acquires Managed DNS Provider Nettica

What Does a DDoS Attack Look Like? (Watch First 3 Minutes of an Actual Attack)

Joining Forces to Advance Protection Against Growing Diversity of DDoS Attacks

Why Managed DNS Means Secure DNS

SPECIAL: Video Interviews from NamesCon 2014 in Las Vegas

Rodney Joffe on Why DNS Has Become a Favorite Attack Vector

Sponsored Topics