|
||
CircleID recently interviewed Meng Weng Wong, the lead developer of Sender Policy Framework (SPF) and founder of Pobox.com. As one of the leading anti spam authentication schemes, SPF is used by companies such as AOL, Earthlink, SAP and supported by anti spam companies such as Sophos, Symantec, Brightmail, IronPort, Ciphertrust, MailArmory, MailFrontier, Roaring Penguin Software, and Communigate Pro. Last month, Microsoft announced its agreement to merge Caller ID, its own proposed anti spam authentication scheme, with SPF—the joint standard is called ‘Sender ID’.
In this two-part interview, Meng Wong explains how SPF got started, where it is today and what could be expected in the future of email. (To read Part I, click here)
CircleID: John Levine, a member of the Internet Research Task Force’s Anti-Spam Research Group, and a popular critic of SPF has been quoted saying:
”...implementing that idea [SPF] would require changes to the Simple Mail Transfer Protocol (SMTP) standard that is the foundation for the e-mail system, and updates to existing mail software packages for every e-mail sender and recipient who want to participate… SMTP has worked the same way for 20 years… If the solution is that we get to change the way SMTP works, there’s a long list of other things we’d like to change about it, too,” he said.”
What is your position regarding this concern?
Meng Wong: He’s right. To explain, I need to talk about forests.
Normally, healthy forests have lightning strikes that lead to fires every few years. Those fires don’t kill trees; the leaves of mature trees are out of reach, so the fires just burn up the dead matter sitting on the forest floor. Small, frequent fires are good for forests. The burned matter reenters the ecological cycle, and new seeds sprout. But wildfires are bad for forests: when fires are suppressed for too long, dead matter builds up for decades, and when it does burn it destroys entire ecosystems.
SMTP is past due for change. If we had been able to change the protocol every three years, maybe spam wouldn’t be such a big problem today. Spam has overrun the internet the way an out-of-control wildfire blazes through a forest. SPF is the first controlled burn in a long time: there will be more to come.
It’s true that changing standards is not easy. But not changing is even worse. There’s a war on—a war against spammers. We have to be quicker to react and quicker to adapt if we want to win it. If things go on, how many years until people just stop using email altogether? They’re not going to give us another twenty.
CircleID: There have been some concerns expressed regarding the incompatibility of SPF with some email forwarding services and websites that use mail-forwarding features. For instance, online greeting card services and news sites use email-forwarding features to allow readers to send email cards and articles to friends. How are such issues being addressed?
Meng Wong: The incompatibility with forwarding and web-generated email is SPF’s single biggest source of fear, uncertainty, and doubt. At first we thought it would be a huge problem. A year later, it turns out that we were wrong. It turned out to be less of a problem than we thought for three reasons.
First, many greeting card sites have actually taken the initiative and voluntarily come into compliance. Contrary to popular belief, these “innocent bystanders” have been monitoring SPF quite closely. They realized that what we asked of them was entirely reasonable, and not that hard to do. One small greeting card site told me that SRS was something they actually wanted to do anyway, to better manage the error messages that go to users. And Evite, one of the biggest “legitimate forgers” of return-paths and “From:” headers, has also changed. Their practices used to cause a lot of problems for spam filters. Last week I got an evite and all their headers were perfect.
Second, the same has been happening for forwarders. Major forwarders like GMX in Germany and Pobox.com in the US have voluntarily implemented the SRS workaround. Other forwarders are sure to follow. (SRS stands for Sender Rewriting Scheme; it encapsulates the original return path so that outbound forwarded mail becomes SPF compliant.) There are three Open Source libraries out there that implement SRS. Forwarders who run their own in-house MTAs are welcome to use any one of them. Folks who run an Open Source MTA such as Postfix, Qmail, Exim, and Sendmail can download SRS patches. In the next few weeks, expect to see packaged MTAs available in RPM, .deb, and similar formats. Those SPF-enabled MTAs will have both SPF and SRS built in and ready to run.
Third, we worked around! Wayne Schlittt is the head of the Nebraska chapter of the SPF community and a major contributor to the SPF effort. In a heroic effort, he tracked down all the major forwarding service providers and put them into a whitelist at trusted-forwarder.org. By default, SPF plugins will query that whitelist. That means that even though forwarders like acm.org and ieee.org have not yet implemented SRS, mail from them will not be rejected.
The vast majority of publishing domains define the strictest form of SPF: their default “-all” means “reject if authentication fails”. And millions of messages covered by SPF records are delivered every day. But how many false positives have been reported to the mailing list? None that I’ve seen.
CircleID: As you are very well aware, Yahoo! has also recently submitted a draft of DomainKeys, its own anti-spam authentication method, to the Internet Engineering Task Force (IETF) to begin its standardization process. How does Yahoo!‘s DomainKeys compare to SPF/CallerID anti-spam authentication method?
Meng Wong: They are complementary and can work hand in hand. SPF and SenderID represent the designated sender school of thought. DomainKeys represents the cryptographic school. Both approaches are valuable. SPF/SenderID is easier to get running, and it is rolling out quickly today. DomainKeys is less mature, more complex, and more long-term. If all goes well, SPF and DomainKeys will meet in the middle and squash spammers like a bug.
In any case, DomainKeys needs SPF/SenderID. SPF is really two things in one: it defines a set of designated sender mechanisms, but it also defines a language in which alternative mechanisms can be specified. Cryptographic mechanisms are the alternatives we had in mind when we designed SPF. If a sender uses DomainKeys, they publish an SPF record that states that fact.
CircleID: Going back to the implementation of the merged SPF/CallerID plan, what sorts of updates or changes are going to be needed by ISPs, email application providers, spam filters, etc.?
Meng Wong: We’re ready to rally the troops.
We have worked long and hard to get agreement from major players, and we have finally built up critical mass.
In the coming months I expect industry to start moving. We’ll be publishing SPF records and upgrading to SPF-enabled MTAs that can implement SenderID and SPF Classic. Forwarders will need to firm up their plans for SRS. ISPs will need to support SMTP AUTH on 587 and start rate-limiting outbound mail servers, and implementing other recommendations seen in the ASTA document. We’re still working out deadlines for “who needs to do what by when” on the spf-deployment mailing list. Industry leaders who want to help set the schedule before the train leaves the station are welcome to join that mailing list.
CircleID: Is there a possibility (and added benefit) for the eventual merging of all these standards? In other words, are we witnessing the initial stages of a collaborative industry move towards the adoption of a single standard?
Meng Wong: The web is a thing with many parts: HTML, HTTP, Javascript, Flash, SSL certificates ... the list goes on. But to the man on the street, it’s all just the World Wide Web.
In the same way, I think that in ten years “The New Email” will be a single thing with many parts: IP-based authentication, crypto-based authentication, reputation services, accreditation services. All of these parts will fit together better than they do today because their rough edges will be worn smooth with use.
And at the borders of email we’ll see gateways to other messaging systems, like Jabber and SMS and video. We might also see things like micropayments and bonded mail (http://www.vanquish.com).
CircleID: Are there any other comments that you would like to share with readers?
Meng Wong: SPF is a success because we followed the Open Source model: we released early and we released often. We listened to feedback. When competing requirements fought, we tried to find a solution that made both sides happy. “Rough consensus and running code” was our guide. Every so often we took a step back to think about what we were really trying to accomplish. When we did that, we usually discovered deep and simple truths that lit up our design from the inside.
I am personally grateful to the hundreds of volunteers who have put thousands of hours of hard work into SPF. It is because of them that we are where we are today: hundreds of millions of SPF-approved messages are sent every day, checked by thousands of MTAs running a dozen different implementations. We’re catching spam and viruses. This stuff is really working.
So now I’d like to ask for money. Changing the world is an expensive job! Flying around the world to get to conferences hasn’t been cheap. The conferences themselves aren’t cheap either. (Expect to pay $500 to attend the San Diego IETF.) According to some reports, spam is a $1 billion problem. It’s costing us all a lot of money. And if we want to fix it, it’s going to cost a little more.
If the work we’re doing is worth money to you or your organization, we ask that you calculate what you would save each month if spam went away, and donate 10% of that number to the SPF project. Yes, we take PayPal.
Sponsored byVerisign
Sponsored byRadix
Sponsored byDNIB.com
Sponsored byVerisign
Sponsored byWhoisXML API
Sponsored byCSC
Sponsored byIPv4.Global
A World-Renowned Source for Internet Developments. Serving Since 2002.
Thank you for sharing an outstanding interview with Meng Wong. I encourage anyone who missed Part I to make sure to give that a read as well.
Though the metaphor is a bit scary, I agree that “There’s a war on - a war against spammers.” The merits of SPF aside, Meng’s “SPF Movement” is pushing people to not only realize, but acknowledge that the problem of spam is not a content issue, but an identity management issue.
While the jury is still out on the impact that SPF alone will have on the problem of spam, we are confident that with the addition of the SPF component, Sender Address Verification (SAV) is without question the most effective solution to stopping spam available today.
The success of SPF is in everyone’s best interest. We at Evite have supported SPF and cleaned up our headers over the past year so that we can be good citizens.
The cost of compliance and diligence is minor compared with the absolute disaster that awaits all of us in a “1980’s SMTP only” world. We at Evite are very excited to be participating in this next generation of email delivery and authentication standards.
Keep up the good work!
so I’m thinking about OpenPGP
explain this to me
if you are going to HAVE to use ESMTP why not add the ability to look up public key for domain ?
if you are doing the domain why not query for user ?
finger server or in DNS record ?
is this in the spec ?
in the future then everyone can use weak crypto for emails and not send everything plain text
(speak to the person in internet cafe or bussiness and they dont understand that their msg is transmited plaintext and maybe through other peoples servers who may or may not read the email )
it would be nice to say you thought of people providing public keys but people dont have to use them…
regards
John Jones
FALSE POSITIVES
When did this interview take place? I’ve posted numerous comments about my SPF false-positive problems to the mailling list, and Meng implimented some new changes to his POBOX service specifically for his customers (including me) who are experiencing this problem!
In a nutshell - any SPF site that doesn’t use “-all” will cause customer emails to be silently erased without warning the sender when the recipient is using Baysian filters (eg: Netscape/Thunderbird etc) and the recipient mail server doesn’t get a SPF “match”.
I have a small lab where I do development for the c=US national directory and identity management project.
Of course, having my SMTP email address out there for years has caused no lack of joe jobbery taking place, so I was pretty excited to implement SPF on my inbound mail server and get a feel for it. With a bit of tweaking it works well. Now at last (along with other polymorphic spam limiting strategies) my filters get a needed rest.
Many thanks to Meng and others for their work on this.
The value of SMTP e-mail has declined due to the fact that it was designed to work without effective sender authentication, a situation that was fairly balanced when corporate and military non-internet e-mail systems had very stringent sender authentication possible with LDAP and X.500 directories holding public keys. Simply put there has been no requirement up til now for regular end users on the Net to have similar levels of identity management that have been available for years.
Forgery of email header identity which was simply a rare inside joke in the early ‘90s, (and rarely abused except for some comic relief), has morphed into a potent destructive force that requires continued work like Meng’s to retain value within one of the most popular protocols.
At this point we need to understand his controlled burn metaphor, and see at what temperature the seeds for a new forest (perhaps an inverted mathematical network naming tree structure?) fall on fertile soil.
Fundamentally this does all come round to identity management. If you think about it, you don’t care where the mail came from, you care that it comes from whom you think sent it. Just a touch of indirection needed. Not where from, but who from. And if that’s a virtual who, than an authenticated virtual who. But “where from” is easier right now.
What constitutes that identity is that the person or entity that sent it is not only who they say they are, but also “acting in role as” or bound by contract to do what they say they can do.
Phishers are exploiting what was intentionally engineered into internet protocols to make them easy and cheap to adopt. Think internet bubble.
They are “bottom fishers” of the Weltanschauung.
Whether we attempt to graft these protective additions like SPF onto an already insecure (but highly popular) system such as SMTP or work from the other direction of being members of some group, like a corporation, commercial walled gardens like AOL or MSN, or a collective identity of a nation state which is the name space where I work, there will be some sort of e-mail convergence as Meng alludes to.
If the result is to balkanize people to use only these walled gardens, defended by Sysadmins, then an essential personal character of edge based communications of the internetwork will have been lost.
The fortress mentality of security is very ‘90s.
What is current is a very po-mo deconstruction of the net to take it back to the personal as well as role based communication where actual relevant connection takes place rather than non-information communication between zombied servers (hosted by clueless end users and going to the same) with broadband connections that is currently occupying a substantial portion of network bandwidth. I don’t have a driveway just so people can dump trash in my front yard.
We are gradually being herded and re-centralized to ISP servers with mail surveillance plug ins, or the relative safety of the corporate domain name and servers. How difficult is it to bring up SMTP? So simple a virus/worm can do it! We don’t need it to be that simple, hence the net is being deconstructed.
If AOL accepts my mail, because I do publish an SPF record, and helps reject mail that does not come from my server to others, that’s a net gain. That’s considerably more fine grained that simply determining that my server has a fixed or dynamic ip address and accepting/rejecting it
> Whether we attempt to graft these
> protective additions like SPF onto
> an already insecure (but highly
> popular) system such as SMTP or work
> from the other direction of being
> members of some group, like a corporation,
> commercial walled gardens like AOL or
> MSN, or a collective identity of a
> nation state which is the name
> space where I work, there will be
> some sort of e-mail convergence as
> Meng alludes to.
>
> If the result is to balkanize people
> to use only these walled gardens,
> defended by Sysadmins, then an essential
> personal character of edge based
> communications of the internetwork
> will have been lost.
This “cathedral” mentality of security is so 60’s. Ok, I suppose that this is the cost of understanding X.500 ;-)
As a former X.500/PKI seller I grew up to the knowledge that SSL should only be used either with the OpenPGP o Kerberos cyphersuites.
Global AND centralized security is a real myth.
Regards.
Conceptually I can see why it might seem that are no applicable examples of globally secure/centralized systems. There are many, and they exhibit differing degrees of centrality, and often redundancy. Sometimes they are “virtually” central, but in fact distributed.
Talking about security in absolutes is not very useful, it’s a process which can be applied in different degrees depending on the requirements.
For most people’s SMTP Internet email; they have not at this point in time required/demanded a high degree of security, but this may be changing since the threats have become more virulent. As Gilmore alludes to, what is the greater problem, the spam, or the solution to the spam?
I don’t think my point that people in general are being herded towards centralized e-mail providers sharing databases, rather than standing up their own servers, as a result of spam is unrealistic. There is discrimination and blacklisting of people’s domains that is patently wrong
Yet spammers are standing up servers on unsuspecting hosts with great frequency.
Just because I can implement what ever is the latest in spam filtering or blocking on my machines, including SPF, doesn’t mean that most people won’t want someone else to do it for them, (and implement many of the suggestions that Meng has outlined).
The same thing may be true with identity, they may realize that whoever is acting as a gatekeeper is not necessarily doing the best job in protecting them from being abused.
People are realizing that “hey I’m only one person, why should I spread my personal contact information all around in each different database, If I can establish one authoritative point of reference, which can be easily changed if need be.”
Case in point is fairly recent activity to legally require accuracy in the WHOIS centroids.
The question is then how to scale out solutions in an ethical/effective manner. There may be good solutions that simply don’t scale, or don’t take hold.
One might define a global system as one that has global usage; rather than defining it as the sum total of all users. Some of these systems are not hooked up to the Internet for obvious reasons.
It’s just that up til now, the need has been more clearly demonstrated and defined for security domains that have been focused on specific corporations, and other institutions, who can and do create security policies that are in turn applied, sometimes in a global manner for their users, but not in terms of the set of all Internet users, who rely on a subset of all protocols, known as Internet protocols. However, some of these protocols can and do rely on higher forms of abstraction, which your message alludes to in ASN.1.
Frequently application programmers don’t adequately understand the protocols leading to glaring security holes, or simply re-inventing the same security problem in a different form.
In terms of end users with high speed broadband connections, there a significant amount of end nodes that are not secure, which argues that simply being an end node does not offer security in and of itself.
There’s no where to run, no where to hide, and an unprotected/unpatched computer hooked up to the net has a life expectancy of less than 20 minutes before it’s turned into a spam zombie or otherwise.
I don’t think the average Internet end user is going to learn C, do regression testing and fix buffer overflows, write their own code, or patch a kernel.
No they are going to go to a highly centralized and controlled source, and download a patch that works until the next public exploit surfaces.
Does the non-Cathedral approach (i.e. the bazaar) work for security for some; sure it does! However, unless one is in a specific security domain, like a company, (and even then it’s difficult) it’s difficult to enforce basic security, but then again that’s what Sysadmins do.
Whether these protocols are adequate is the case in point. Frequently, end users may require more security, and then they can layer on whatever suits them. But specifically, there is a lack of verifiable identity in SMTP.
How one chooses to layer in identity, it’s still the same problem, since security problems are global in nature. One attempt to abstract those problems is found in the Common Critera.
http://www.commoncriteriaportal.org/
This ill-designed web form ate my reply.
I’m sorry I won’t write again my 4567 characters reply.
What kind of web-incompetents designed this site?
You can contact me at .(JavaScript must be enabled to view this email address) if you find this conversation interesting, as I do ;-)
Regards.