A revolution is taking place on the Internet, with new sites redefining how we interact online. The next-generation Internet is emerging in collaborative and interactive applications and sites with rich, varied media (images, video, music). As with many revolutions, this one is driven by the younger generation, which is adopting social networking sites like MySpace and video sharing sites like Google’s YouTube. But the general shift is not restricted to the young, as more mature consumers and businesses alike are exploring the possibilities of collaborative, media-rich applications.

This major shift in Internet applications has its unintended victims. One of them turns out to be the Domain Name System (DNS). This service is part of nearly every Internet interaction, and operators of major IP networks know that DNS is essential to service performance and availability.

The Internet is poised for its next major burst of growth and usage as billions of telephones, fax machines and PDAs join the desktops, laptops and servers already communicating on the Internet. This is a critical moment for traditional telephone carriers and Internet Service Providers (ISPs) developing next-generation voice, data and multimedia services; missteps or slowdowns can be enormously costly in the race for market and mindshare.

The next-generation Internet applications, as represented by today’s MySpace and YouTube, substantially increase the DNS query load for carriers. As is explained below, each page visit to one of these sites may require ten times more DNS queries than for traditional websites. As subscribers and traffic continue to grow to these sites, the underlying DNS traffic grows at a much more rapid rate. And as DNS traffic reaches capacity, the network slows down for all applications and becomes vulnerable to attacks and malicious traffic.

This issue affects all broadband providers, including DSL providers, cable operators, and wireless/cellular carriers offering IP services. Maintaining service performance and availability in this rapidly-changing world will require re-architecting the DNS infrastructure with much more capacity to sustain the new generation of Internet users and uses.

The Social Bottleneck

There are many traditional reasons why broadband carriers are seeing sustained growth in DNS traffic, including increases in broadband subscribers and increased usage per subscriber. This kind of ‘organic’ growth is to be expected, and is part of any carrier’s capacity planning. But broadband operators worldwide are experiencing a rapid growth in DNS traffic that cannot be attributed to subscriber growth. While viruses, worms or attacks cause temporary spikes in DNS traffic, these changes are significant and enduring. For example:

• A North American provider measured DNS traffic growing 8% per month.
• A large European provider found that in the past year their DNS traffic grew 92% while their subscriber base grew 15%.
• Another large Europe provider reported that their DNS traffic doubled in the last six months. Previously, it doubled every year.

Something else is going on in addition to increases in subscribers and growth in online usage. The next-generation of web applications is one of the key factors, and the story of MySpace is a good example.

The Social Slowdown

MySpace is an extremely popular social networking site. MySpace users post profiles of themselves, to which their friends add in a continuous blog. Its collaborative nature is representative of the Web 2.0 movement.

A typical MySpace profile page is a rich assortment of images and blogs posted from friends. Users can post videos and flash-based content, as well as links to favorite songs in MP3 files. In most cases, each of these content pieces is stored in a separate DNS domain. For example, each image belonging to a friend is retrieved from a distinct URI. This means that retrieving and displaying a profile page may require hundreds of DNS lookups in the background—compared to ten or so lookups for a ‘standard’ B-to-C web page.

MySpace is one of the most visited sites on the Internet. Each of those page downloads may account for ten times or more the amount of DNS traffic of a typical web page visit. Here is an important clue to the recent, unusually high increase in DNS traffic. And, alas, there is more to the story than meets the eye.

The Content Distribution Conundrum

The use of content delivery networks (CDN) puts further strain on the DNS for rich media and social networking site queries.

Increasing numbers of websites use content delivery (or content caching) networks to distribute content on the Internet. Content delivery companies use globally distributed servers to deliver rich content and interactive applications most effectively to users around the globe. They do so by caching content locally, and then using the DNS to direct users to the best source for content, based on current performance, load, proximity and availability information. This capability has been critical to helping popular websites deliver a good experience to users across the world.

Content delivery companies want to maintain control over how traffic is directed across their global platform and dynamically redirect traffic to the most appropriate servers in real or near real-time. This is accomplished by setting short Time to Live (TTL) values on the DNS data they deliver for low-level links in pages. This means that caching name servers cannot maintain the data in the cache for long to answer subsequent queries for the same pages, which puts further strain on the DNS system.

Back to our example of the MySpace page, assume that twenty kids from a high school all want to check out the latest MySpace posting from their favorite musician, and all use the same Internet provider. Not only does each page download generate hundreds of lookups, but in many cases the caching name server cannot use cached data and must return to the global Internet to get the right DNS data. Not only does the site increase the DNS traffic, it reduces the effectiveness of the DNS server’s cache and increases the time the server spends waiting for responses from “upstream” authoritative DNS servers.

The improvements that content distribution networks provide to the Internet at large are well worth the trade-off of increased DNS traffic, which is very small in volume in comparison to image and video content. By distributing data around the Internet in the most efficient manner (and caching frequently-accessed data closer to the user/s), content delivery networks both improve response times for users and reduce the amount of data traveling through core IP networks.

However, this is small comfort to the DNS administrator struggling to handle DNS traffic doubling every year with increasing query latency.

A sign of things to come: the Web 2.0 bottleneck?

The MySpace example above is not an isolated problem—it is the harbinger of things to come as new uses of the Internet take hold.

The term Web 2.0 refers to this next-generation of more collaborative websites rich in media: podcasts, social networking sites, web applications and wikis among them. YouTube, the popular video sharing site acquired by Google in late 2006, is another example of collaborative, media-rich Internet usage that is wildly popular, relying on content delivery networks and increasing the load on the DNS.

If the DNS impact of social networking and Web 2.0 sites only affected users of those sites, the problem would be still be serious. Wireless carriers, for example, want to gain subscribers from the demographic that uses MySpace and FaceBook. As wireless connectivity increases, wireless users will take advantage of their ‘instant connectivity’ to use location-based social networks. Slow page downloads will reduce the value of the service.

But the ramifications go well beyond the social networking users.

An overloaded DNS server eventually slows down query responses to all queries—affecting all other services and users. Carriers have to continuously build out DNS capacity to provide adequate response times. And, as systems run closer to capacity, the network is more vulnerable to Denial of Service attacks or to DNS overloads due to traffic generated by viruses and worms.

For carriers that still rely on general-purpose Berkeley Internet Name Domain (BIND) software for DNS, the growth of DNS load puts them in an impossible situation. A BIND server can only handle so much DNS traffic before it starts behaving erratically, dropping packets, requiring frequent restarts as well as more hardware. And the more servers you add, the greater your ongoing cost of operation. Both CAPEX and OPEX increase—eroding the cost benefits of using open source software.

To make it more complex, none of this is happening in isolation. This server proliferation takes place while carriers are trying to streamline and optimize the IP infrastructure to support network convergence. Triple- and quadruple-play services are adding voice, video and mobility to the IP infrastructure, increasing service level expectations and traffic. And emerging standards like DNSSEC will put further load on the DNS.

A DNS Infrastructure for Web 2.0

Carriers need to shift how they think about the IP naming/addressing infrastructure of their networks, and DNS in particular. DNS has long been an ‘invisible’ service that performs in the background. Many carriers have relied for many years on BIND software, which was designed as a general-purpose DNS server (both authoritative and caching) for enterprises and small-range carriers. However, in a Web 2.0-enabled world, carriers trying to protect the customer experience cannot neglect the DNS.

Instead, they need to make a commitment to building a carrier-grade DNS infrastructure that can handle the rapid growth in traffic without sacrificing performance and resiliency. To do so, they will need to use the most efficient DNS solution possible that maximizes load capacity while minimizing server resources.

A good guideline is to provide enough DNS capacity to handle up to three times current peak load.

In addition, the DNS infrastructure must be tightly integrated in the entire network management scheme for optimal availability and reduced operating costs. It needs carrier-grade monitoring and alerting, and robust online configuration and remote management capabilities.

Carriers are not alone in this DNS investment. VeriSign, which operates two of the Internet’s root servers, has recently announced a massive investment in its DNS infrastructure, and is in the midst of increasing capacity 10,000-fold by year 2010. The Domain Name System as a whole has proven itself tremendously scalable over the past two decades, during a period of enormous change in IP networks. It will continue to scale and grow to handle the next-generation of Internet and converging IP networks, as long as we give it the attention and care it deserves.