Home / Blogs

DPI is Not a Four-Letter Word!

As founder and CTO of Ellacoya Networks, a pioneer in Deep Packet Inspection (DPI), and now having spent the last year at Arbor Networks, a pioneer in network-based security, I have witnessed first hand the evolution of DPI. It has evolved from a niche traffic management technology to an integrated service delivery platform. Once relegated to the dark corners of the central office, DPI has become the network element that enables subscriber opt-in for new services, transparency of traffic usage and quotas, fairness during peak busy hours and protection from denial of service attacks, all the while protecting and maintaining the privacy of broadband users.

Yet, DPI still gets a bad rap. Guilty until proven innocent! Why is that?

DPI means different things, because it is an overloaded term. I can think of at least four separate product categories of DPI:

  1. Traffic Management: DPI that classifies application traffic by examining the headers, without looking into the actual content itself.
  2. Surveillance: DPI that logs, reconstructs, or plays back communication exchanges.
  3. Ad-Insertion (and profiling): DPI that profiles subscriber web browsing or search activities, inserts cookies, or logs URLs visited by a subscriber.
  4. Security: DPI that examines content for viruses, trojans, or other forms of vulnerabilities.

Paramount to each of these product categories is privacy. Service providers and consumers share in concerns over privacy, as do industry luminaries. Yesterday, according to ZDNet, Sir Tim Berners-Lee, "inventor" of the World Wide Web, spoke out against the use of deep packet inspection citing concerns over how snooping on clicks and data reveals more information about people than listening to their conversations.

His concerns are valid. And I can attest, having worked with service providers around the globe, that service providers are deeply aware of how important it is to protect consumer privacy. That is why service providers are becoming more transparent and giving consumers choices with opt-in and opt-out capabilities. This new era of transparency is as much a result of consumer interests, service provider best practices, and increasing regulatory pressures, as it is an indication of the broader shift of how DPI-based services are being used.

That is why Phorm, the targeted advertising service company mentioned in the ZDNet article which uses DPI, has a technology that can't know who users are and allows users to switch it off or on at any time (opt-out or opt-in).

But transparency and consumer opt-out are not limited to broadband service providers and DPI. Yesterday, Google launched "interest-based" advertising on their partner sites and on YouTube, where ads will associate categories of interest based on the types of sites you visit and the pages you view. And, in line with DPI and service provider models of transparency and consumer choice, Google is offering transparency, choice with Ads Preference Manager, and a non-cookie based opt-out capability.

So at the heart of any service over broadband, not just DPI-based services, is the need for transparency, fairness, consumer choice and protection while preserving the privacy of individuals. These are the new discussion points that need to transcend specific technologies in the network. The public debate and regulatory directions has to be centered on these key areas (stay tuned as Arbor becomes more active in these arenas).

As for DPI itself, it has proven to be a critical network element in service provider networks, by providing those things that we all hold dear: privacy, protection, fairness and transparency. DPI is not a four-letter word!

By Kurt Dobbins, Chief Technology Officer, IP Services, Arbor Networks

Related topics: Access Providers, Broadband, Net Neutrality, Networks, Policy & Regulation, Privacy, Telecom


Don't miss a thing – get the Weekly Wrap delivered to your inbox.


I'm a bit confused as to how Scott Francis  –  Mar 13, 2009 10:47 AM PDT

I'm a bit confused as to how exactly a third party eavesdropping on a conversation provides privacy (or protection, fairness or transparency). Perhaps you define these terms differently from the average network end-user?

However, I can certainly see how DPI can provide revenue opportunities, and LEO cooperation opportunities, and media industry monitoring opportunities, and ever more over-subscription opportunities. (why build out capacity when you can simply degrade performance for users, protocols or content deemed "unacceptable"?)

The era of ubiquitous transparent encryption for all traffic between endpoints can't come soon enough.

This is what happened to me yesterday. Joe xx  –  Mar 13, 2009 12:27 PM PDT

This is what happened to me yesterday.  I woke up to find I could not send e-mail and I eventually discovered port 25 was blocked by Comcast.  After going through several customer support calls to my cable company I was told port 25 was blocked.  They would not tell me the exact reason but they suggested I was either a spammer or a botnet had taken over my computer.  They told me they would release the block one time but if it happened again the block would not be removed.

Since i want to know what happened I reviewed the Comcast privacy policy and I found that I am supposed to able to review the information they have on my account.  I requested the information they collected so I could review it.  Suddenly the privacy policy does not matter and they claim the information they collected was "proprietary." Their privacy policy also says Comcast does not read your e-mail.  So what information do they have?  what did I do?  All i know is that if I do it again they will block my port.

I filed a complaint with TRUSTe so i can see what they have.

All ISPs block outbound port 25 Joe George Ou  –  Mar 16, 2009 2:01 PM PDT

All ISPs block outbound port 25 Joe and they've been doing it for many years.  At one time before Comcast blocked outbound port 25 for anything other than Comcast's SMTP server, Comcast's customers were unknowingly sending ~25% of the world's spam.

Yet this has nothing to do with DPI and it's not some kind of evil plot.  Consumers can still do emailing through Comcast or third party email providers.  Those third parties such as Gmail use port 465 for SMTP over SSL.

Why do you confuse port 25 filtering with dpi, Joe XX? Suresh Ramasubramanian  –  Mar 13, 2009 8:35 PM PDT

I am probably wasting my time asking this ..

Indeed you are Richard Bennett  –  Mar 16, 2009 6:04 PM PDT

But it's noble of you to try to spread a little light.

Suresh and Richard Joe xx  –  Mar 16, 2009 7:21 PM PDT

I read some of your other posts.  When someone disagrees with you all the both of you do is hurl insults, call people trolls, etc. and never argue the issues.  This is a common problem with network manager types.  I will not respond to either one of you again until you can provide an answer that is "smarter than fifth grader."

Because Comcast is inferring in their network Joe xx  –  Mar 14, 2009 6:50 AM PDT

Because Comcast is inferring in their network management policies that they use DPI in order to trigger Port 25 blocking.  Upon reviewing their labyrinth of policies their is actually conflicting information.  In some parts they say they use DPI tools without reading your e-mails directly, in other parts they say they may actually read the e-mails.  In this case I really don't know if they used DPI or not and they won't tell me.  The point is that I can't find out even though they claim their practices are transparent and they say I have right to know what info they have about my account.

Joe, you're sadly mistaken George Ou  –  Mar 16, 2009 3:31 PM PDT

You don't use any kind of DPI technology to do simple port filtering which has been around for around two decades.

one more time Joe xx  –  Mar 16, 2009 7:12 PM PDT

It is what triggered the port blocking.  If it was not DPI then you are saying they read my e-mail (either directly or via a complaint from a third party) or is there some other possibility?

Joe, one last try George Ou  –  Mar 16, 2009 7:52 PM PDT

Joe, one last try.  There is no DPI technology involved here.  It's simple port blocking which is around 2 decades old. I and other people on this site have tried to correct you and explain that this has nothing to do with DPI or behavioral or interest based advertising.  I've and others have explained to you that external port 25 is blocked to stop spam and that this doesn't prevent you from using external SMTP email servers.

No human at your ISP reads your emails Joe.  The anti-spam and anti-virus systems inspect your email for spam and protects you from annoying and malicious emails.

First you say no DPI is involved. Joe xx  –  Mar 16, 2009 8:01 PM PDT

First you say no DPI is involved. 

Then you say "No human at your ISP reads your emails Joe.  The anti-spam and anti-virus systems inspect your email for spam and protects you from annoying and malicious emails." That is a DPI system.

Which is it?  DPI or not?  How do you know what triggered Comcast to block my port? 

You keep saying I am wrong but you never address the issue or answer the questions.  This is what happens in these discussions.  If I was discussing any other port there would be a rational discussion instead of the "religious" ports that have to do with e-mail.

Joe, don't confuse port blocking with anti-spam or anti-virus George Ou  –  Mar 16, 2009 8:16 PM PDT

Joe, don't confuse port blocking with anti-spam or anti-virus.  All three systems work independently of each other and they don't need the presence of the other two systems to work.

Everyone here is trying as patiently as possible to explain to you that Comcast systematically blocks all users from getting to the Internet on TCP port 25 and that this is very normal behavior for consumer broadband accounts.  Just about every major broadband provider uses this technique to avoid being the world's leading source of spam.

Your line of questioning is getting beyond ridiculous at this point, and there's not much more I can add to this.

wrong Joe xx  –  Mar 16, 2009 8:38 PM PDT

You are wrong.  According to their written policy they block port 25 on a case-by-case basis based on a security incident.  They have a chart of the ports they block.  Some ports are blocked for all users but 25 is only blocked based on security incidents.  Their privacy policy says I can have the information associated with my account and I have the opportunity to dispute it.  They won't Give it to me.  If their policy said they blocked all port 25's or if their policy said they will not give me access to my account info then there is no issue. 

Most network admin types never give any thought whatsoever to these issue and they won't even discuss it.  They think they are the judge and jury on how their network operates without any regard to legal issues, the fact that they have a contract to provide a service to the general public, etc.  Anyone who tries the issue is ridiculed by many system admin types and the discussion turns into insults.  The person is labeled as a spammer or an idiot and disregarded.  This is how the Facebook people acted when they recently changed their privacy policy.

Once again you simply say my questions are "ridiculous" without adressing the issues.  This is the religion of anti-spammers and network admins, not a network discussion of the tradeoff between functionality and security.

# 18 Reply (max. reply level reached)  |  Link  |  Report Problems
Comcast Policy confusion Joe xx  –  Mar 15, 2009 7:15 AM PDT

Comcast Privacy Policy states:

a.  We will not read your outgoing or incoming e-mail… We also monitor the performance of our Service and your Service connection in order to manage, maintain, and improve the Service and your connection to it. We (or our third party providers) use tools to help prevent and block "spam" e-mails, viruses, spyware, and other harmful or unwanted communications and programs on the Service. These tools may automatically scan your e-mails … and other files and communications in order to help us protect you and the Service against these harmful or unwanted communications and programs. However, these tools do not collect or disclose personally identifiable information about you…

For Residential Services states:

a.  … [Y]ou acknowledge and agree that Comcast and its agents have the right to monitor, from time to time, any such postings and transmissions, including without limitation e-mail,…

Spam Policy states:

a.  Port 25 is conduit on a computer that spammers can take control of and use to send their spam - often without the user ever knowing his/her computer has been "hijacked", and

b.  Comcast works with our customers to block access to Port 25 and protect their PC.  Comcast recommends that our customers establish a more secure email configuration on their PC - Port 587 - We have made it easy by creating a one-click fix that automatically configures your computers to this safer PC configuration. 

Customer Privacy Notice states:

a.  You may examine and correct, if necessary, the personally identifiable information regarding you that is collected and maintained by Comcast in our regular business records.

Acceptable Use Policy for High-Speed Internet Services states:

a.  … Comcast uses reasonable network management tools and techniques to protect customers from receiving spam and from sending spam (often without their knowledge over an infected computer)… and

b.  Comcast reserves the right to investigate suspected violations of this Policy, including the gathering of information from the user or users involved and the complaining party, if any, and examination of material on Comcast's servers and network. … You expressly authorize and consent to Comcast and its suppliers cooperating with … system administrators at other Internet service providers or other network or computing facilities in order to enforce this Policy.

Network Management FAQ states:

a.  Will the technique target P2P or other applications, or make decisions about the content of my traffic?

No. The new technique is “protocol-agnostic,” which means that the system does not manage congestion based on the applications being used by customers. It is content neutral, so it does not depend on the type of content that is generating traffic congestion. Said another way, customer traffic is congestion-managed not based on their applications, but based on current network conditions and recent bytes transferred by users.

It's like this, Joe Richard Bennett  –  Mar 16, 2009 6:09 PM PDT

There is something in the world called Spam, most of it sent from computers that have been infected by things called "bots." When the volume of your outgoing mail gets to be quite high compared to the average user, a responsible ISP will check some of this e-mail for spammy content. This isn't exactly "reading" your mail - there;s not a group of sys admins sitting around laughing at you for the way you write, there's a program that pattern matches your e-mail and gives it a spam score. If the score is high enough to determine that it is spam, they shut down port 25 to stop the spamming.

This is a good thing to do, but I agree with your that the Terms of Use for that particular ISP could be more clear. In any event, spam prevention is a good thing.

First of all the people posting the Joe xx  –  Mar 16, 2009 7:10 PM PDT

First of all the people posting the arrogant insults, like you and Suresh, does not add to the discussion.

The evenutal port filtering is not the issue.  The issue is what caused the port to be filtered.  Did Comcast use DPI or not?  I cannot tell and Comcast won't tell me so the providers are not doing what they say.  You claim you know but you really don't know what happened.  Was it DPI, a complaint from a third part, or simply some type of error (which is what I suspect)?  You seem to insist it is and George says it is not.  Maybe it was and maybe it wasn't.

Something caused the filtering.  I only send out a few personal e-mails.  If I am a botnet i need to know to take corrective action.  If not, then what happened?  They told me the port would be unblocked one time as a courtesy but if it happens again it won't be unblocked but they won't tell me why they blocked it.  So now what do i do, sit around guessing?  I don't use Comcast e-mail so I would need a port forwarder to use e-mail.  That is not something i can do quickly when I wake up in the morning and can't send e-mail.

There is no need to talk like a third grader as I understand how the spam system admins work.  Checking content of spam and deciding "spammy content" is reading e-mail.  Using DPI without a human reading it is not necessarily reading the e-mail but it is error-prone.

Comcast just went through a large legal process with Network Neutrality and has a TRUSTe-endorsed policy.  This is not some small time error.  Blocking ports is not protocol-agnostic.  Saying they don't read e-mail in some parts and saying they do in other is also not a small error. 

"Spam prevention is a good thing" is a Motherhood statement that doesn't address any issue.  Comcast blocked a service in the name of security.  They only give a workaround for those using Comcast services, not external ones like I use.  The issues are haphazardly blocking services in the name of spam prevention and using security as an excuse to block competing services.

Don't use port 25 Richard Bennett  –  Mar 16, 2009 7:33 PM PDT

Personally, I use port 465 to send e-mail from my Comcast account at home and 993 to receive it, so any use of port 25 on my home router would be a solid indication of a bot infection. As I don't know what goes on inside your home network (neither do you, apparently,) that's all I have to go on. I'm a Comcast customer with no e-mail blockage problems.

So rather than whining about a perfectly rational practice on the part of your ISP and imagining sys admins are reading your personal mail, why don't you secure your system and setup your e-mail in a responsible way? Comcast will give you a copy of McAfee for free to help get you started.

Your postings remind me of the old Joe xx  –  Mar 16, 2009 8:23 PM PDT

Your postings remind me of the old newsgroup days.  I am a CISSP and understand the issues both technical and political.  You are in your own little network management world and do not understand all the issues.  This stuff is for the general public, not a small group of arrogant admins like yourself who think they know everything when their knowlege is limited to niche areas.

Then you use your Salem Witch Trial techniques:  "you were blocked so you must be guilty" and "it didn't happen to me so there must be something wrong with you." then pile on insults.  These arguments are childish and have no place here.

Port translation can be done but i dodn't have time to set up in the 10 minutes I had to send an e-mail.  Besides, the RFC standard says Port 25.  But of course there must be something wrong with anyone who doesn't do what you do.  They must be guilty, incompetant, stupid, and a troll.

The issue is that Comcast has to give me the information they collected on my account, whether DPI, reading e-mail, or whatever.  I drafted a lawsuit and their legal department wants to set up meeting.  In any case, it is clear they are posting conflicting policies no matter what happened to me.  We will see what they say about why I was blocked and whether DPI was used.

Hilarious Richard Bennett  –  Mar 16, 2009 8:39 PM PDT

You don't have time to check the SSL button in Thunderbird (or the equivalent) but you do have time to draft a meritless lawsuit?

Damn, you must write fast.

The lawsuit has nothing to do with Joe xx  –  Mar 16, 2009 9:06 PM PDT

The lawsuit has nothing to do with the actual blocking of any port.  it has to do with a written policy that says they will give me account information and now they are withholding this information.  Can you describe what is meritless about that?

You see how it works is that companies provide false publkic information.  Then that gives fuel to people who want more regulation.  Then a bunch of mindless regulations get passed and a bunch of bureaucrats get hired to enforce it.  Then there are big conferences all over the world where people like you argue with people like them and never get any work done.

If companies would "do what they and say what they do" then normal market forces are at work. 

If you would read the post instead of thinking up stupid insults you would see that Comcast did unblock the port, they just won't say why they blocked it, which is the issue you won't address.

Practicalities The Famous Brett Watson  –  Mar 16, 2009 10:21 PM PDT

Joe, I think you are going to save yourself a lot of time and aggravation by avoiding this issue rather than tackling it head on. It seems perfectly likely to me that Comcast are not upholding their promises in relation to information disclosure, but I'm a firm believer in Hanlon's Razor here: they are not being evil or devious, they are just a large bureaucracy which is incapable of internal consistency by merit of simple size. You could take the matter to the courts if you so desired, but that seems like an awful lot of time, effort and money, given the best possible outcome is what? You get to know why they're blocking you? Or maybe you think it will be a worthwhile ideological victory? You're far less cynical than I am if that's the case.

Were I in your position, here's what I would do: observe best current practices for message submission (BCP 134, RFC 5068) and send email to a smarthost via port 587. That way you get the satisfaction of being a technical purist, and Comcast's beliefs about your use of port 25 become irrelevant. Yes, it's a little annoying that you can't use port 25 even though it is a technically reasonable approach, but port 25 hasn't been the gold standard for mail submission in a decade or so, and a technical purist needs to keep up with the times.

So the thought I would like to leave you with is this: if you were observing best current practices, this would not be a problem for you. Solve the problem at your end, not Comcast's: it's the path of least resistance and fewest ulcers.

The blocking was probably not devious, just Joe xx  –  Mar 16, 2009 10:57 PM PDT

The blocking was probably not devious, just some error that caused the blocking.  It is devious when they don't give access to the info that caused the blocking.  They are collecting information, maybe DPI, maybe reading e-mail, who knows.  It leads to blocking competitors, snooping on users on behalf of RIAA, incompetant network management, etc.  The network people who aren't really involved in these things will just dismiss it and ridicule anyone who brings it up (until the company is caught red handed).

For $200 I can file a suit to prevent their privacy policy from being posted or distributed.  They don't have to give me the info, they just can't post a policy that says they will.  I did the Opt-out for binding arbitration when Comcast introduced it on 2004 so I can take the matter to court.  Basically, all I need to do is sit back and watch then try to explain to the judge what all these policies mean .  It should comical.  "we don't read e-mail", "we sometimes read e-mail", "Our DPI system is protocol agnostic", "we block ports based on security incidents and DPI" " we give customer's access to their account info ... except for this guy because he actually asked for it."

The reality is my e-mail server cannot use other ports.  I need to upgrade it to get SSL or I need to set up port translation.  I can drop what i am doing becasue some admin decided to block the port.  You see the communications of the customer paying bill doesn't matter, what matters is keeping the system admin happy.

To post comments, please login or create an account.

Related Blogs

Related News

Explore Topics

Dig Deeper

IP Addressing

Sponsored by Avenue4 LLC

Mobile Internet

Sponsored by Afilias Mobile & Web Services


Sponsored by Verisign

DNS Security

Sponsored by Afilias

Promoted Posts

Buying or Selling IPv4 Addresses?

ACCELR/8 is a transformative IPv4 market solution developed by industry veterans Marc Lindsey and Janine Goodman that enables organizations buying or selling blocks as small as /20s to keep pace with the evolving demands of the market by applying processes that have delivered value for many of the largest market participants. more»

Industry Updates – Sponsored Posts

Verisign Named to the Online Trust Alliance's 2017 Audit and Honor Roll

Attacks Decrease by 23 Precent in 1st Quarter While Peak Attack Sizes Increase: DDoS Trends Report

Leading Internet Associations Strengthen Cooperation

i2Coalition to Present Tucows CEO Elliot Noss With Internet Community Leadership Award

Michele Neylon Appointed Chair Elect of i2Coalition

2016 U.S. Election: An Internet Forecast

MarkMonitor Supports Brand Holders' Efforts Regarding .Feedback Registry

Verisign Releases Q2 2016 DDoS Trends Report - Layer 7 DDoS Attacks a Growing Trend

Verisign Named to the Online Trust Alliance's 2016 Honor Roll

Verisign Q1 2016 DDoS Trends: Attack Activity Increases 111 Percent Year Over Year

Mobile Web Intelligence Report: Bots and Crawlers May Represent up to 50% of Web Traffic

Dyn Weighs In On Whois

Data Volumes and Network Stress to Be Top IoT Concerns

Verisign Mitigates More Attack Activity in Q3 2015 Than Any Other Quarter During Last Two Years

Protect Your Privacy - Opt Out of Public DNS Data Collection

Dyn Evolves Internet Performance Space with Launch of Internet Intelligence

Introducing Verisign Public DNS: A Free Recursive DNS Service That Respects Your Privacy

Verisign's Q2'15 DDoS Trends: DDoS for Bitcoin Increasingly Targets Financial Industry

Protect Your Network From BYOD Malware Threats With The Verisign DNS Firewall

Verisign Named to the Online Trust Alliance's 2015 Honor Roll