With advancements in hardware and software, sophisticated filtering technologies are increasingly being applied to restrict access to the Internet. This happens at the level of both governments and corporations. For instance my company, Renesys, is headquartered in the “Live Free or Die” US state of New Hampshire. In our small town of roughly 10,000 folks, we know of a local company who tries to restrict non-work related (e.g., shopping) websites from their employees. Unfortunately, someone who works there can’t read about Amazon’s cloud computing as a result—a small bit of collateral damage. Entire countries act in much the same way. The OpenNet Initiative keeps track of such state-sponsored restrictions and publishes interesting maps based on the level of filtering by topic. But given the open nature of the trust-based Internet, one country’s restrictions, if not handled very carefully, can easily foul the global Internet nest we all live in. This blog is about one such story of Internet restrictions in China becoming visible (seemingly at random) from other parts of the world and going undetected for 3 weeks. Given the increasing complexity of this technology, the difficulty in controlling a very open Internet, and the strong desire of some to do just that, this could be a harbinger of things to come.

Background

To understand this problem, one needs to consider Internet routing, the behavior of DNS and the root nameservers, and the economics of Internet transit. So let’s briefly review a few basics before we describe the incident. When you type

www.facebook.com

into your browser, your computer must first contact a DNS server to convert this name into an IP address in order to contact the host serving this content. Answers to DNS requests are cached on both your machine and the various servers involved to reduce the load of subsequent identical queries. Now suppose for the moment that the caches on your machine and your DNS server are both empty and you make the above query. Your DNS server first contacts a

root nameserver with your request. If configured according to convention, the root nameserver will not provide the answer to your query, but instead direct your DNS server to the .com nameservers. In turn, those will direct your DNS server to the Facebook nameserver, which will ultimately provide an IP address to one of Facebook’s web servers. At least that is how it is supposed to work.

Now suppose organization C runs a root nameserver and C doesn’t want you going to Facebook. Nothing requires C to direct you to the .com nameservers. Since C sees your complete request, it could just answer you directly. If C gave you the wrong answer, it would effectively block your access to Facebook, assuming you didn’t know enough to pick a different server. Since the Internet runs on trust, you’ll also end up caching C’s invalid response (known as cache poisoning) with C being the one who tells you how long to cache the result! (Alternatively, C could actually provide the correct answer, but a firewall in front of C could alter the result on its way back to you. Either way, you lose.)

Incident Details

While this scenario might seem very unlikely, it in fact just happened with the

I-root instance run out of China, as first reported by Mauricio Vergara Ereche in Chile and subsequently blogged on here. In fact, the problem existed from March 3^rd until March 25^th, before it was reported on and corrected. Despite the fact that a lot of people could have been impacted, the chances of any one of them having gotten the incorrect DNS response are extremely remote, as we’ll explain later. This is thanks again to the way DNS operates and the overall resiliency of the Internet.

So let’s review exactly what happened here. First, as is well known, China censors the Internet in a variety of ways. One way is to return invalid answers to DNS requests to Chinese users. For example, at this moment, a Chinese DNS server is returning 46.82.174.68 as an IP address for www.facebook.com, when in fact, all legitimate Facebook IPs are of the form 66.220.x.y or 69.63.x.y. Seemingly random, but often unrouted, IPs are also returned for www.twitter.com, www.youtube.com and many other domains. This is normal and expected behavior inside of China.

However, China also happens to house an instance of a root nameserver, the I-root, and when this server became visible outside of China on March 3^rd, anyone who happened to query it could have gotten bogus responses. Keep in mind that there are 13 different root nameserver IP addresses (associated with the A-root, B-root, ..., M-root) and the I-root is just one of these, namely, 192.36.148.17. In addition, there are dozens of instances of the I-root housed in many locations around the world. To get a bogus DNS response outside of China, you not only had to query the I-root, you had to query the Chinese version of it. The countries with the most number of network prefixes that could have queried the Chinese I-root during this time are given in the following graphic. Not surprisingly, the most exposed countries were all in Asia, but some some prefixes in the US were also vulnerable, more than half of which geo-locate to California.

So let’s review the unlikely series of events that would have been required to observe a bogus answer to www.facebook.com.

1. You attempt to go to www.facebook.com.
2. You don’t have this entry in your DNS cache, nor does your DNS server.
3. Your DNS server does not have the .com servers cached either.
4. Your DNS server happens to choose the I-root (as opposed to A-root, B-root, C-root, etc).
5. Due to current Internet routing in place at your location, your DNS server happens to be directed to China’s instance.

Since Facebook is blocked in China, your DNS server does not get the expected list of .com servers, but rather a bogus response to your original request, either from the I-root itself or a firewall in between. You cache the result and can’t “friend” anyone for a while.

Think for a moment about how unlikely all of this is. Responses for .com nameservers are set to expire only after 48 hours. So to have any chance of seeing a bogus response, the first request into your DNS server after its cached .com entries expire must be for a blocked domain in China and

your DNS server has to query the I-root, one of 13 possibilities. (Once the .com servers are cached, there is no need to query to the root servers for them.) In addition, of the many instances of the I-root, you have to somehow end up selecting the Chinese one. But with 1.7 billion Internet users surfing the web for 3 weeks, it was bound to happen and be reported.

Of course, you don’t really have any control over which I-root instance you see from your location. That is determined by Internet routing. Many of the root nameservers are anycast from multiple locations around the world. This means that the associated IP prefixes are announced from multiple locations, all of which house servers with copies of the appropriate data. BGP, the Internet routing protocol, is then used to sort out who sees which instance of the root servers from which locations. In general, the Chinese I-root instance is only visible from within China, but for 3 weeks these routes leaked out to the global Internet. This announcement exited China when it was leaked by the China Internet Network Information Center (AS 24151).

The careful reader might wonder how anyone in the US could have selected the Chinese I-root. Isn’t there a closer one? Well, that’s the thing about Internet routing. It’s driven more by economics than by physical distance, although the two are often related. For example, suppose two smaller providers, call them A and B, agree to exchange traffic with each other for free. This common arrangement on the Internet is known as peering

and allows A and B to save money, specifically, transit costs to larger providers. Suppose further that A (or one of its customers) is running the I-root. If B needs to get to the I-root, it should pick its settlement-free peering link with A, rather than its link to a larger carrier for whom they have to pay. China Telecom, by far the largest carrier in China, peers with nearly 100 other providers. If those providers or their customers aren’t running an instance of the I-root themselves, they might use their peering link to China Telecom to reach their instance. This is how countries far from China could end up selecting the Chinese I-root as the “best” of many possibilities.

Conclusions

This story illustrates both the fragility and the resiliency of the global Internet. It’s fragile because it is ultimately trust-based and almost anyone can violate that trust, deliberately or by accident (and there is no reason to think this wasn’t an innocent mistake). It’s resilient because there are often many alternatives or workarounds for any screw-ups or attempts at control. The Internet is a big place full of very clever people—making it tough to wall off.

Update Mar, 31 2010: Graph figure updated for further clarification.