Home / Blogs

The Future of Interdomain Interconnection and Traffic Control

Nick Feamster

Today, we are excited to announce, in partnership with the Open Networking Foundation and Open Source SDN, the official launch of iSDX — open-source controller software for an industrial-scale Software-Defined Internet exchange point. iSDX allows independently operated networks to interconnect and exchange traffic in completely new ways. This software, which we've been developing for nearly three years, now finally operates at the scale of the world's largest IXPs and interoperates with SDN-capable hardware switches, opening up new possibilities for interdomain business relationships and traffic exchange.

For the past three years, a team of researchers at Princeton, led by Arpit Gupta, has been developing mechanisms that could fundamentally change how independently operated networks interconnect and exchange traffic. It is well-known that independently operated networks on the Internet — often called Autonomous Systems (ASes) — are interconnecting and exchanging traffic in vastly different ways than they did even ten years ago. The first formal observation of this evolution was Labovitz's Internet traffic report in mid-2010, which observed that the structure of interconnection relationships which once formed a relatively strict hierarchy has become decidedly less structured, as large "hypergiant" content providers and content distribution networks gain increasing leverage in peering relationships. The figures below, from the BITAG report on Internet interconnection from 2015, show how this hierarchy has evolved.

The hierarchical structure of interdomain relationships in the 1990s. (Source: BITAG / Click to Enlarge)

The flatter interconnection structure of today's Internet. (Source: BITAG / Click to Enlarge)

Unfortunately, while Internet interconnection has undergone rapid and dramatic restructuring, the underlying protocols that support interconnection have not fundamentally changed since the 1990s — the last substantial update to the BGP standard, RFC 4271, was effectively to standardize the long-ossified route selection process. And, while there have been some changes to the protocol (e.g., the introduction of route reflectors, 32-bit AS numbers, etc.), none of these things have fundamentally changed the way that ASes might be able to exchange traffic or establish business relationships.

Imagine if instead networks that interconnected could make decisions on how traffic is exchanged based on the application type; we could have more complicated business relationships, such as "application-specific peering". Or, suppose that networks could change their selection of routes to neighboring networks dynamically, based on (say) whether a certain link was experiencing high traffic volumes, a denial-of-service attack, or a traffic ratio imbalance. Or, suppose that the network switches could dynamically enforce more complex interconnection agreements that could meter and shape specific application flows (e.g., streaming video), without affecting how other traffic flows are prioritized or routed.

We believe Software-Defined Networking (SDN) may hold the keys to some of these new capabilities, and over the past three years, we have been working to build a platform, an industrial-grade software-defined Internet exchange point (iSDX), that can both support these pilot use cases and explore new ones. This project augments a standard Internet exchange point (IXP) architecture, whereby participant networks exchange routing information through a shared route server, to avoid pairwise BGP sessions. We have augmented the "stock" route server to incorporate an SDN control platform based on Ryu that allows participants to (1) use BGP routing as the mechanism to exchange topology; (2) incorporate specific SDN policies that allow participants to override the default forwarding behavior that BGP would otherwise provide.

An SDX controller allows each participating network to both (1) exchange BGP routing information, as in conventional route servers; and (2) express specific traffic exchange directives using SDN programs that the controller composes into a single coherent set of forwarding rules. The controller installs the rules into the switch.

Although we initially presented this idea in Summer 2014, it has taken several years to develop the SDX codebase so that it can operate in industrial-scale settings. One of the major challenges was scale: accepting full BGP routing tables from each participating AS will result in hundreds of thousands of routes that must be translated into forwarding rules in a hardware switch. Unfortunately, most commodity SDN switches cannot support anywhere close to a forwarding table of that size. Over the past year, we have developed ways to compress these tables by several orders of magnitude, which now brings all of the ideas that we envisioned in the initial SDX design much closer to reality.

The paper that Arpit Gupta is presenting today at the USENIX Symposium for Networked Systems Design and Implementation (NSDI) which is a joint work with Laurent Vanbever at ETH Zürich and Marco Canini at UC Louvain, explains all of these optimizations in detail. Today, we are excited to announce the launch of the iSDX open-source project, in collaboration with the Open Networking Foundation's Open Source SDN project. The long-running open-source project has been undergoing steady development over the past few years. Over the past year, we have worked closely with partner enterprise networks to harden the code and test it on hardware switches, including the low-cost Quanta LY2. Now, we are excited to welcome the broader community to our open-source effort, through a partnership with the Open Networking Foundation. The Open Source SDN community site has launched, as has a new mailing list for the iSDX open-source community.

Although today's announcement represents the culmination of several years of work to-date, we believe that this partnership with the Open Networking Foundation will enable an exciting set of new opportunities. We look forward to supporting iSDX and exploring a much broader range of applications, modes, and business relationships for Internet interconnection, ranging from better traffic management and control, better support for interdomain security on the Internet, a much wider range of possible interconnection agreements and business relationships, and many other applications and use cases that we haven't even yet imagined.

By Nick Feamster, Professor at Princeton University

Related topics: Broadband


Don't miss a thing – get the Weekly Wrap delivered to your inbox.


Prof. Feamster,Several of the pilot use cases Frank Bulk  –  Mar 16, 2016 4:54 PM PDT

Prof. Feamster,

Several of the pilot use cases you listed would seem to violate the FCC's Net Neutrality rules, as they are application specific.  Is that your understanding, too?

Differentiated Treatment Nick Feamster  –  Mar 16, 2016 5:20 PM PDT


Good question. I think the short answer is that "it depends".  Here's a longer writeup explaining various use cases where differentiated treatment of traffic constitutes "reasonable network management", which the OIO allows for.


That's the quickest answer I could give to a question that concerns a 400+ page document!


Thanks for replying. I scanned through Frank Bulk  –  Mar 16, 2016 6:15 PM PDT

Thanks for replying.  I scanned through that report and I felt a more cautious recommendation.  While the "reasonable network management" is allowed, choosing which applications get preferenced (or de-preferenced) is not.
No Throttling: broadband providers may not impair or degrade lawful Internet traffic on the basis of content, applications, services, or non-harmful devices.

[oops] replied below Nick Feamster  –  Mar 16, 2016 6:51 PM PDT

[oops] replied below

Differential != Degraded Nick Feamster  –  Mar 16, 2016 6:50 PM PDT

Great question. Reasonable network management can sometimes entail differentiated treatment of traffic. Quoting the BITAG report, "When differentiated treatment is applied with an awareness of the requirements for different types of traffic, it becomes possible to create a benefit without an offsetting loss. For example, some
differentiation techniques improve the performance or quality of experience (QoE) for particular applications or classes of applications without negatively impacting the QoE for other applications or classes of applications." There are many examples of this, largely because different applications have different latency, throughput, and loss requirements.  Happy to discuss more offline.

Working with IXPs? John Souter  –  Mar 17, 2016 1:15 AM PDT

Which Internet exchanges have you engaged with in doing this research?

To post comments, please login or create an account.

Related Blogs

Related News

Explore Topics

Dig Deeper

Afilias Mobile & Web Services

Mobile Internet

Sponsored by Afilias Mobile & Web Services


Sponsored by Verisign

DNS Security

Sponsored by Afilias

Promoted Posts

Now Is the Time for .eco

.eco launches globally at 16:00 UTC on April 25, 2017, when domains will be available on a first-come, first-serve basis. .eco is for businesses, non-profits and people committed to positive change for the planet. See list of registrars offering .eco more»

Industry Updates – Sponsored Posts

Ofcom Benchmarking UK Broadband Performance Welcomed, But Needs Considerable Improvement

Neustar Chief Technology Officer Appointed to FCC's Technological Advisory Council

Neustar Labs Innovation Center Grand Opening (Video)

How Secure is Your Mobile Network? And Does It Even Matter? (Webinar)

Neustar and University of Illinois Launch the Neustar Innovation Center

Nominum Launches World's First Purpose-Built Suite of DNS‐Based Solutions for Mobile Operators

72 Confirmed Talks - If You're Attending, Now is the Time to Register

eComm 2009: Discussions on Restructuring Global Telecoms

Ben Scott and Free Press in the Network Age

Supernova Interview: David Isenberg

Wendy Seltzer Interview: How Law Impacts the Network Age

Jon Peha, Chief Technologist, FCC, on the National Broadband Plan

Supernova Interview: JP Rangaswami

Expanding Internet Access Driving Software Piracy, Study Says

eComm Programme Guide Now Available

eComm 2009 Event Next Week: Defining the Post-Telecom Era

Ribbit Co-Founder Crick Waters to Deliver Keynote at Inaugural Emerging Communications Conference

eComm Conference to Showcase Leading Communications-Industry Innovators

Wireless Innovation to Be Center Stage at Emerging Communications (eComm) Conference 2008