|
||
|
||
On Saturday Aug 7th, DNS provider DNS Made Easy was the target of a very large denial of service attack. As far as can be determined the total traffic volume exceeded 40 Gigabit/second, enough to saturate 1 million dialup Internet lines. Several of DNS Made Easy’s upstream providers had saturated backbone links themselves. There are indications that not only DNS Made Easy suffered from this attack, but the Internet as a whole.
An attack on DNS is an attack on the Internet in two ways. Name servers are a critical point in almost every Internet access. But as our research shows, the consequences of this attack were wider than the attack’s primary target.
According to DNS Made Easy, service impact was limited. According to our measurements it was around 5-10% on a global basis.
“In some regions there were no issues, in other regions outages lasted a few minutes, while in other regions there were sporadic (up and down) outages for a couple of hours. In Europe for instance there was never any downtime. In Asia downtime continued longer than other regions. In United States the west coast was hit much harder and experienced issues longer than the central and east coast.”
DNS was designed from the ground up to be resilient to individual server failures. In theory this should make the loss of a few servers irrelevant. On top of this, the provider has implemented an anycast routing infrastructure, which works to ensure that DNS queries all over the world are resolved regionally. Note that because of the anycast routing of this provider, outages are related to the location where the clients (resolvers) are located, not the servers whose names are being queried.
However, measurements/analyses that I made in collaboration with WatchMouse.com have uncomfortable implications. WatchMouse regularly measures the performance, including the DNS resolve time, of thousands of sites, through a network of more than 40 stations spread over all continents.
In a dataset with sites whose DNS records were served by the provider, resolve times rose from a normal average of less than 100 milliseconds to over 200 milliseconds in the hours of the attack. Average failure rates in this dataset are around 1%. During the attack hours, this rose to 5% and even 10%. As can be expected, these failure rates differed greatly by monitoring station, though it is hard to see a geographical pattern.
Another dataset consists of regular measurements of more than 300 sites, with a total of more than 300.000 individual measurements over a period of 8 days. In contrast, none of these sites had their DNS service from DNS Made Easy. These sites are operated by a wide variety of industries.
On the seven days leading up to the attack, the daily average DNS resolution time in this dataset was between 352 milliseconds and 379 milliseconds. On the 7th of August, the average was 453 milliseconds, which is a significantly higher. Averaged by the hour, resolution times rose to 600 and even 800 milliseconds. There are failure rate fluctuations in this dataset, but they appear to be uncorrelated to the attack.
Note that these measurements support the provider’s claim of shorter resolve times. A regular DNS lookup takes 350 milliseconds, but DNS Made Easy’s average is less than 100 milliseconds.
In conclusion, these results are disturbing because even sites that are totally unrelated to DNS Made Easy were affected in their response times. The implication of this is that this denial of service attack was big enough to have collateral damage on the rest of the Internet.
Sponsored byDNIB.com
Sponsored byVerisign
Sponsored byWhoisXML API
Sponsored byIPv4.Global
Sponsored byCSC
Sponsored byVerisign
Sponsored byRadix
A World-Renowned Source for Internet Developments. Serving Since 2002.
Link to data?
Yes, larger Internet outages can have downstream effects and an attack of the claimed magnitude would probably be demonstrative. Typically, outages are also met with a general grousing and media articles in relevant publications about sites being down and unavailable (smaller attacks in the past have). Internet analytic companies like Pingdom or Renesys typically weigh in with slowness or route instability. None of those occurred and it seemed like an average day on the Internet.
It’s not comprehensive but here’s a starting point: http://twitpic.com/2f2rde I don’t see the 5%-10% slowness on end user look up times for domains that we serve. Outages and large scale attacks are scary things that none of us hope to ever have. I think back to the SQL Slammer worm and the derivative outages that caused. I think we can agree that something happened (either observed or self reported) but the magnitude and downstream effects are still open for debate.
Looking forward to comparing datasets.
Yes can we get a link to your data? DNS Made Easy made claims to have best industry times & when pushed for specifics much faster than you seem to indicate.
@David, which data would you like to see in particular?
@Jeremy: 5-10% refers to domains served by DNS made easy, not the internet as a whole. Nevertheless, our measurements do show above average resolve time slowness.
Depending on your point of view you could say that only a 40G attack was noticeable, of that even with a 40G attack, the victim still managed to serve 95% of its customers.
@Rick: average speeds are very dependent on the geography served/tested. If you have a set of locations and a number of DNS providers we might have a look at relative speeds.
@peter—I just think it’s odd that there was no talk amongst major network operators about this attack and various peering exchanges don’t show any traffic increases across their links at the time of the attack. It’s certainly possible that this all happened over PNIs and avoided all public fabrics, but for something that made other networks hurt, I would have expected to see some more evidence of it actually occurring.
40gbps is a large attack by anyones metrics, even if directed at a widely anycasted address.
Peter van Eijk’s research findings make for an interesting read. The methodology for the collection and analysis of the data was well explained and seems to have been done in a scientific manner.
Just in case some poor soul stumbles across the comments on this article and mistakenly believes that a collegial discussion regarding data analysis is occurring here… Let me dispel that notion with some full disclosure:
- Mr. Ulevitch runs OpenDNS and formerly ran EveryDNS (before it was acquired by Dyn Inc.).
- Mr. Hitchcock works for Dyn Inc. (DynDNS / Dynect).
- Mr. Rumbarger is a former employee of Neustar - UltraDNS.
The above comments are not from ‘disinterested’ parties. In fact, these commenters represent competitors of DNS Made Easy. These comments appear to be a part of their campaign to downplay the attack against DNS Made Easy. Why they would do this is anyone’s guess. Logically, I would think that these DNS providers would be touting their own capabilities to defend against an attack of the magnitude that targeted DNS Made Easy. Instead they have attempted to downplay the attack. I can only conclude that this strategy was adopted because they could not defend themselves against an attack of the magnitude of the one which targeted DNS Made Easy.
Here is some more disclosure - I work for DNS Made Easy. For the record, no one employed by DNS Made Easy has had any contact of any sort with the author of this article and we have shared no data with him. I stumbled across this article just like everyone else did.
As stated in his article, Peter van Eijk made his measurements in collaboration with Watchmouse. Watchmouse is a customer of DNS Made Easy and uses our DNS service for the domain watchmouse.com (a public fact that can easily be determined using any number of methods). I wasn’t personally aware that watchmouse.com used our DNS service until I checked after reading this article. DNS Made Easy was completely unaware that this measurement and monitoring of our services was occurring or would be used in an article. DNS Made Easy, however, continues to invite any and all responsible members of the press or potential customers to monitor our services and present fair and unbiased reports of our service performance in any forum.
It looks to me like the author has a very reputable background in his field and is more than capable of analyzing data sets of this type. I would personally enjoy seeing a education/resume/capabilities comparison between any of the previous commenters and the author of this article. A quick Google search shows that Peter van Eijk has a Masters degree in Computer Science and over 30 years technology experience in places like AT&T;Bell Labs, CVI (Dutch Rail Automation)/EDS, EUNet, Deloitte & Touche, and other unnamed organizations as a technology consultant. In particular the author appears to have broad experience in networking/internetworking, stress tests, and performance studies. Any of the previous commenters willing to compare their education/resume/experience with the author’s?
I don’t know anything about the data that was collected beyond what is presented in this article, but it would be my guess that the data can’t be released without fully anonymizing it. I would also guess that anonymizing the data would reduce its value greatly. Perhaps these interested parties could explain how they are more qualified to analyze this data than the author who is a highly experienced and disinterested third party.
We monitor our network internally and externally using a variety of tools and I don’t have any disagreement with the author’s findings regarding our service given the tools and methodology used in his tests.
David Miller
CTO - DNS Made Easy
David -- You are right, I could not deal with a 40gbps attack today. I'm more interested in knowing if you really did have a 50gbps attack (or 40) because I don't really believe I've heard of attacks that big on the Internet. If it's true, and there is some evidence of it, I'd like to see it so I can know that I need to make my pipes and DDoS defense systems even bigger than they already are. I have no ill will towards you or DNS Made Easy or anyone else. DDoS sucks, and we all have to deal with it. But I'd like to see as much information about it as possible, so I can act accordingly.
Our letter to our customers revised our estimate of the attack to have ranged from 20-40Gb/s over time. DNS Made Easy Letter to Customers Our letter to our customers also described the methods that we used to determine these levels. Unfortunately, we cannot provide any further information. I know that this doesn't provide the solidity that you would like and I apologize for that. Attacks well above the size of our attack have been reported. Akamai publicly states that they have absorbed attacks of up to 200Gb/s. Akamai Security Capabilities Prolexic states that they have protection available in excess of 150Gb/s. There is likely a reason for this. The warnings are out there. Take a conservative estimate of the number hosts for a large botnet and do some simple math and you quickly get up to very scary numbers.
David, no need to be defensive. I think what you are seeing is the industry rallying together and wanting to learn more about the event. Events of this size get people’s attention because any of us could equally be a target and we’re all in it together. I’m also not trying to hide my identity, I think everyone on Circle ID has some idea of who I am and where I work. It’s also on my member page.
Love to propose a call or something that we can all do to mitigate and defend against these things (similar to the Inside Baseball/DNS roundtable event we hosted with 18 people from 12 companies in the DNS space in May). Might make sense to set something up in the fall before next year’s event.
Jeremy Hitchcock (CEO of Dyn)
.(JavaScript must be enabled to view this email address)
603-296-1554
Currently travelling with limited access to my files. I will follow up in a few days.
I appreciate the constructive comments on my post and I feel proud to be a member of this community dedicated to making a better Internet.
Some methodology clarification.
The measurements used in the research are the result of long running studies. The ‘independent’ dataset has no relation in any way with DNS Made Easy’s services, to the best of our knowledge. It is monitored for the purpose of measuring website performance across industries, and does no measurements of DNS Made Easy at all. The ‘provider’ dataset contains websites whose DNS records were served out DNS Made Easy. Again, the primary purpose of this dataset is to monitor website performance. DNS measurements are a collateral benefit of these measurements, so to say.
I am putting up some graphs of the measurements and slightly more detail on my blog at petersgriddle.net, as circleid.com does not appear to accommodate graphs.
As for publishing the raw data, there appear to be mostly practical objections to that. The data basically consists of measurements made on the public internet. Contact me privately for details.
Some more disclosure. I am a paying subscriber of DNS Made Easy’s services as well as a number of other DNS providers. I control most of the domainnames in the ‘provider’ dataset, but none of the domains in the ‘independent’ set. I have collaborated with Watchmouse on a few client engagements, but am otherwise independent.
Presumably those using DNS Made Easy as a provider of secondary DNS servers would still have had a functional (if slower) DNS during the attack, unless their domain was specifically targeted.
That raises the issue of motivation, which hasn’t been reported on. The twitter feed notes they identified “the domain”, which might give some clue as to motivation.
If large DNS providers are seeing DDoS attacks frequently it would make sense for more critical domains to configure third party services as secondaries, or run an additional secondary. Thus ensuring their domain doesn’t have a shared fate with the DNS providers name servers.
Occurs to me that if one has more name servers than strictly needed for redundancy, one could place each domain on fewer than all of them randomly (e.g. 4 of 6 or 3 of 6) or following some permutation, and thus reduce the worst effect of these attacks (assuming they are targeting a specific domains name servers) to only those who are unlucky enough to have the same set of name servers. Although this potentially trades performance for robustness in a relatively rare situation.
3 of 6 would mean only a sixth of hosted domains shared the same set. Others would lose at most 2 name servers during a similar DDOS (assuming all name servers are on separate physical infrastructure, ignoring Anycast, etc etc).
Interesting approach. See for a more comprehensive story about additional DNS DOS countermeasures: blog.easydns.org
Your DNS provider, if any, can implement most of these on a larger scale.
Note therefore, that if you take this approach, you are basically saying that you are smarter than your provider. Why would you then employ them?
If you are the victim of an attack, who can fight it better? You or your provider? If you are collateral damage, you may be able to get out of the firing line, but you still run the risk of stumbling in the process.
“If you are the victim….”
Peter I think you miss the point, one of their customers was a target, many of them suffered. This approach avoids having a shared fate with all the other customers. The providers could do something similar as I outlined, but clearly in this instance they weren’t, and whatever route the providers take it is unlikely to scale as well.
If your domain is the target you are probably hosed anyway, since it typically requires far less fire power to DDoS web, email and other services. Indeed DDoS on DNS would seem a particularly hard difficult route to take to getting anyone off the net, unless it is perhaps Amazon or Google (i.e. someone with substantially more Web and Email infrastructure than DNS).