Home / Blogs

An Interview with the Lead Developer of SPF - Part I

CircleID recently interviewed Meng Weng Wong, the lead developer of Sender Policy Framework (SPF) and founder of Pobox.com. As one of the leading anti spam authentication schemes, SPF is used by companies such as AOL, Earthlink, SAP and supported by anti spam companies such as Sophos, Symantec, Brightmail, IronPort, Ciphertrust, MailArmory, MailFrontier, Roaring Penguin Software, and Communigate Pro. Last month, Microsoft announced its agreement to merge Caller ID, its own proposed anti spam authentication scheme, with SPF — the joint standard is called 'Sender ID'.

In this two-part interview, Meng Wong explains how SPF got started, where it is today and what could be expected in the future of email.

CircleID: A lot of attention has been paid to Microsoft's recent announcement about an agreement to combine Caller ID, its anti-spam authentication method, with Sender Policy Framework (SPF), another popular anti-spam authentication scheme. Can you share with us some background about your involvement with SPF?

Meng Wong: In 2002 Paul Vixie, the brains behind BIND, wrote a short paper titled "Repudiating Mail-From”. That inspired two other proposals, "Reverse MX” by Hadmut Danisch and "Designated Mailer Protocol” by Gordon Fecyk. In late 2003 I combined the best of both proposals and called the result "SPF". (I picked the acronym first and decided what it stood for second, which is why old references call it Sender Permitted From. Now it means Sender Policy Framework, which is a touch more accurate. Yes, references to sunblock are intentional.)

I presented the SPF concept at YAPC and (O'Reilly Open Source Convention) OSCON in 2003. Audiences liked the idea, and the mailing list kept growing. In November 2003 at the Hackers Conference I met Mark Lentczner, my co-author and an absolute genius at protocol design. During the conference and in the following weeks we rewrote the spec together. Eric Raymond talked about SPF at the MIT Spam Conference in January and I presented it before the IETF in Seoul in March.

My role in the SPF community is Chief Communicator: I run the website and go to conferences to spread the word. I also wrote Mail::SPF::Query, the reference library.

CircleID: Who is using SPF today and what level of success has it achieved in the fight against spam?

Meng Wong: In the six months since we declared a spec freeze, 20,000 domains have self-reported at the online registry; that's probably a fraction of the true total. Lots of domains have published records, including AOL, Amazon, Google, O'Reilly, SAP, TicketMaster, Mail.com, w3.org, Earthlink and Verizon. And the ones who haven't published are working on it.

We expect adoption to pick up exponentially; according to some estimates, the number of sites checking SPF doubles every three weeks. SPF plugins are available for all the major Mail Transfer Agents (MTAs).

CircleID: SPF, as we understand it, is aimed at addressing a fundamental weakness in Simple Mail Transfer Protocol (SMTP). Could you give us some insight on how this anti-spam authentication works? How does it differ from that of Microsoft's Caller ID?

Meng Wong: Suppose you get a letter in the mail. The return address on the envelope says "Amazon.com". You know that Amazon is based in Seattle, so if the postmark says "Seattle, WA" you could believe it's really from Amazon. But what if the postmark says "Nigeria"? That's suspicious.

The key insight in Vixie's original paper was this: domains publish MX records to tell senders where to send incoming mail. Domains can also publish records to tell receivers where to expect outgoing mail to come from. This kind of thing is generally called a Designated Sender Scheme, because domains designate their outbound mail servers. RMX, DMP, SPF, and CallerID are all Designated Sender Schemes.

Here's how it works:

Suppose example.com receives mail at 192.0.2.1 and 192.0.2.2. When it sends mail, it uses those two servers, as well as 192.0.2.3. Example.com would publish an SPF record that said: "v=spf1 ip4:192.0.2.1 ip4:192.0.2.2 ip4:192.0.2.3 -all".

When a mail server gets mail that claims to be from someone at example.com, that server can fetch example.com's SPF record and see if the connecting SMTP client is designated.

There are a number of shorthands available: for example, "v=spf1 mx" means "I send mail from my MX servers".

Vixie's paper talked about running the authentication check against the domain of the MAIL FROM parameter. Its descendants RMX, DMP, and SPF all use the return-path identity.

Some of the proposals that were presented to the IETF use different identities: instead of the return-path, those proposals attempt to authenticate the HELO argument, or the PTR of the client IP address. The SPF community has recently developed a Unified SPF model that embraces all of the above semantics.

CallerID defines an algorithm to extract from the headers what it calls a Purported Responsible Address. The PRA starts out as the "From:" header, but if there's a "Sender" header present, "Sender" will override. If "Resent-From" is present, that overrides Sender. The algorithm was developed empirically; it is meant to pull out the last logical hop from the headers. If a message was forwarded through a service like acm.org or pobox.com, the PRA should be the alias at the forwarding service that was responsible for causing the final delivery.

SenderID applies SPF checks against the PRA. SPF Classic applies SPF checks against the return-path. In both cases, though, the DNS record that's used in the checks is generally in SPF format. That's why I sometimes talk about "SPF records" and "SPF checks", even when the authenticated identity is different from the return-path.

CircleID: What are the advantages of having the two methods merged? What are the key measures that are required in order to make this collaboration successful?

Meng Wong: In May, Microsoft agreed to adopt SPF's syntax and semantics for their Caller ID proposal. The result, which we're calling SenderID, applies SPF checks to the Purported Responsible Address from the message headers. I believe it represents the best of both worlds. The best thing about the CallerID proposal was the use of XML, which provides a path to future extensibility. SenderID uses a dual-syntax model: because the core semantics are drawn from SPF but are expressed in XML, SenderID clients will be able to parse both XML and regular SPF records. This means that if, in the future, the SPF syntax runs out of steam, and we need to say things that can't be said in the SPF language, we can upgrade to XML with a minimum of pain.

Meanwhile, SPF Classic, which applies SPF checks to the return-path, is still going strong. It is more mature and is better understood; it continues to enjoy a lot of support in the open community. Many people on the MARID mailing list have argued that SPF Classic and SenderID can and should proceed in parallel, especially since they both use the exact same records but can be used for two mutually compatible purposes.

CircleID: You have been quoted in the media, saying:

"In the last six months or so, there's been a fair amount of uncertainty (about sender authentication). These are two very similar proposals and do many of the same things. The big players have been telling us 'When you get your story straight, we can go ahead,' but until that happens, people have been waiting to see what happens"

What sorts of issues have caused the above mentioned uncertainties, particularly among the"big players"?

Meng Wong: The big players wanted CallerID and SPF to converge because they were both designated sender schemes. It didn't make sense to have two different syntaxes and two different semantics for two different identities, because receivers would have to code up two sets of functions that do basically the same thing, and publishers would have to publish two records that say the same thing. Now that Microsoft has committed to remain backward-compatible with the SPF syntax, publishers don't have to worry about publishing both, and receivers don't have to worry about parsing both.

Read Part II of this Interview.

By CircleID Reporter

Related topics: DNS, Email, Security, Spam

WEEKLY WRAP — Get CircleID's Weekly Summary Report by Email:

Comments

Re: An Interview with the Lead Developer of SPF - Part I Tal Golan  –  Jun 29, 2004 9:59 AM PST

Thank you for sharing an outstanding interview with Meng Wong. I very much look forward to reading Part II and beyond.

Meng's work and ability to garner support within the industry is amazing. While SPF alone does not solve the problem, it is without question the best step forward, from a global infrastructure standpoint, that has been made to date. I applaud Meng for accomplishing a feat that many thought was impossible.

I would like to add that Sendio, Inc. (http://www.sendio.com), provider of the world's only licensed Sender Address Verification (SAV) anti-spam appliance, has incorporated SPF. In the future, please include Sendio in your list of companies in support of SPF.

While the jury is still out on the impact that SPF alone will have on the problem of spam, we are confident that with the addition of the SPF component, Sender Address Verification (SAV) is without question the most effective solution to stopping spam available today.

Re: An Interview with the Lead Developer of SPF - Part I Paul Vixie  –  Jun 30, 2004 6:41 AM PST

I'm surprised and pleased to learn that the SPF folks were aware of my work (MAIL-FROM) in this area. I would like to correct two misimpressions, and share an observation.

First, MAIL-FROM always used what Meng Weng calls the "return path", see section 2.3 of my proposal where it refers to RFC2821 (which describes the envelope, not the header) and the pseudo-code which refers to "MAIL FROM:<>" which is an envelope ("return path") artifact. In fact, the very name of my proposal ("MAIL-FROM") pertains to the SMTP verb (MAIL FROM) used to express what Meng Weng calls the "return-path identity".

Second, this idea is not original with me. I finally wrote it up and identified the corner cases in 2002, because I was cleaning out my inbox and found mail which Jim Miller had sent me describing the basic idea in 1998. I very much wish that I had jumped on this in 1998 when I first heard about it, and that the community had implemented this before Y2K, so that e-mail could be nearly forgery-free by now. Instead, the IETF ASRG is still trying to merge the merged CallerID/SPF proposal with some other competing work, and the final result is very much an open issue.

Finally, an observation. After co-founding MAPS and turning that crank for a number of years, I know that spam is "like a drug" in that people will go to almost any lengths, no matter how absurd, to send more of it. No "designated sender scheme" will ever be able to cut down the amount of spam that's sent, or received. All it can do is help domain holders avoid the brand dilution of having their domain name forged by spammers. This is a valuable contribution, but we must make it clear that none of these schemes will stop or even slow spam, and that their benefits accrue to domain holders, not to spam recipients.

To post comments, please login or create an account.

Related Blogs

The Resolvers We Use

A Brave New World or Do We Need to Discuss IT and Ethics?

ccTLDs Might Be Property

The EFF and Hanlon's Razor

Domain Name Abuse Is a 4 Letter Word

Related News

Topics

Industry Updates – Sponsored Posts

Join Paul Vixie & Robert Edmonds at the Upcoming Distinguished Speaker Series

Q3 2014 DDoS Trends: Attacks Exceeding 10 Gbps on the Rise

LogicBoxes Announces Automation Solutions for ccTLD

3 Questions to Ask Your DNS Host About DDoS

Introducing Our Special Edition Managed DNS Service for Top-Level Domain Operators

Afilias Partners With Internet Society to Sponsor Deploy360 ION Conference Series Through 2016

Neustar to Build Multiple Tbps DDoS Mitigation Platform

The Latest Internet Plague: Random Subdomain Attacks

Digging Deep Into DNS Data Discloses Damaging Domains

New gTLDs and Best Practices for Domain Management Policies (Video)

Nominum Announces Future Ready DNS

Non-English "IDN Email" Addresses Are Finally Working!

New from Verisign Labs - Measuring Privacy Disclosures in URL Query Strings

Video Interviews from ICANN 50 in London

DotConnectAfrica Delegates Attend the Kenya Internet Governance Forum

3 Questions to Ask Your DNS Host about Lowering DDoS Risks

Continuing to Work in the Public Interest

Verisign Named to the OTA's 2014 Online Trust Honor Roll

4 Minutes Vs. 4 Hours: A Responder Explains Emergency DDoS Mitigation

Dyn Acquires Internet Intelligence Company, Renesys

Sponsored Topics

dotMobi

Mobile

Sponsored by
dotMobi
Afilias

DNSSEC

Sponsored by
Afilias
Verisign

Security

Sponsored by
Verisign
Minds + Machines

Top-Level Domains

Sponsored by
Minds + Machines