A minor point of disagreement: the SPF “-all” declaration is not the same as null MX. The SPF “-all” declaration means, as you say, “this domain does not ever send mail”. The null MX, on the other hand, means “this domain does not ever send or *receive* mail”. I have configured a domain to act as a spam-trap, and it has the SPF “-all” declaration to assist others in blocking the domain when it is randomly used by spammers as a sender address. It would not be possible to configure this domain with null MX and continue to use it for its intended purpose.

I must acknowledge the minor point Brett suggests - but, in any case, i typically wouldnt want to accept mail from a domain to which i couldnt deliver bounces to.  such bounces would only clutter up space on my mail queues, and occupy resources that would be better used in processing other, more valid, email.

Along with Joe’s excellent paper on zombies cited in this article I would suggest reading this - http://darkwing.uoregon.edu/~joe/port25.pdf

I no longer support SPF either. For my lastname domain, instead of simply removing the SPF records, I put in some interesting active commentary - slowly resolving records*.  For customers, I did what you did - removed the SPF records.

*dig elvey.com TXT to see.

For those who suggest that widespread implementation of email authentiation, along with appropriate certification and reputation systems is the way to stop UBE, one question:

Will this approach prevent the use of infected computers to spew streams of spam into the Internet?

The simple answer is No. I agree that email authentication, along with certification and reputation systems are part of the solution, but the first priority is network security.

As to SPF, which version? Version 1 or 2. If you are speaking of SPF version 1, which implementation?

The working group established by the IETF to write a DNS based standard using the IP address was closed by the IESG in October, 2004.

Why? A number of reasons, depending on one’s perspective. But in my view the real reason was quite simple.

Not enough technical vetting of the Sender ID Framework and little real world large scale experience with any of the protocols to respond to the various concerns.

The upshot? In closing MARID, the IESG asked for submission of individual proposals for consideration as experimental protocols. At the same time, the IESG instructed that a focused technical directorate be established to review the drafts.

The purpose? The IETF was proposing to sponsor a wide scale experiment. Prior to conducting this experiment, it was felt necessary to asses the proposals technically to determine whether any of the protocols would cause deleterious harm to the email infrastructure, if implemented on a wide spread basis.

Last fall, the Sender ID Framework editors submitted drafts. In December, after re-organizing itself, the SPF version 1 editors submitted a revised draft.

Since December, much work has been done by the IESG and the various directorates set up to review these drafts.

The SPF version 1 editors are now proposing to submit a revised draft.

The goal? To clarify certain issues. Among other things, the revised draft: (i) removes DNS zone cuts; (ii) recommends: checking the domain in HELO/EHELO before checking the domain in the SMTP mail from command; and (iii) recommends against using the PRA algorithm found in the Sender-ID Framework to check SPF version 1 records.

At the same time, some in the SPF community want to stipulate that the experimental nature of SPF version 1 ended in December, 2003 and to call upon the IETF to consider SPF as an RFC standard and not as an experimental protocol.

My point? The IESG in closing down MARID in October, 2004 felt there was need to conduct a large scale experiment under supervision of the IETF of DNS based email authentication using the domain and IP address.

I appreciate many were upset with the closure of MARID and some feel the IETF made itself irrelevant in the field of e-mail authentication.

People want to stand up and say, “do this now and we can end the problem.” However, authenticating email is not easy. The email infrastructure is complex and in many cases quite fragile.

The good news? While the IESG is doing its work, the experiment has already begun. Businesses that do bulk mailings are being encouraged to publish policy records, while members of the Message Anti-Abuse Working Group (MAWG), lead by AOL have been testing the various versions of the sender policy framework.

In fact, Andrew Newton, former co-chair of the MARID working group, recently submitted the first version of an information draft to the IETF reflecting the observations of MAWG titled Considerations for the use of the Sender Policy Framework. This work will aid the community at large in understanding the practical problems with implementation and allow the community to make informed choices about how best to proceed.

My own suggestion? Having read this report, perhaps a large service might want to suggest to those using its feedback loop that implementation of CSV records is also obligatory, allowing it to test the HELO/EHELO scope as the basis for IP based email authentication using the CSV design, while working with Yahoo! and Cisco on the design and implementation of a light weight cryptographic method of message header authentication.

John Glube
Toronto, Canada

John

Thanks for the recap of spf / other authentication protocols’ history with the ietf

A strange thing is that most of the current spf records published by people are not for the purpose of ensuring authentication .. but for an interesting reverse application of spf that aol has said they’ll implement. 

If you want to get whitelisted (as in, not subjected to automated filters) by aol, and want a feedback loop from aol for complaints generated by AOL users about mail from your IP space to be routed back to you, set up spf records so that AOL can get your IP space from them.

So, you have a whole lot of senders, ranging from good to bad to downright ugly, as well as a whole lot of other sites that just want feedback loops from AOL, going out and publishing spf records [hopefully with a ?all or ~all - given the small minority of sites unwise enough to reject on spf checks it is not that great a problem yet, but..]

spf records for whitelisting does mean AOL’s postmaster staff has much less work than if people opened tickets and had AOL staff manually / semi automatically update the whitelist, and (what some people don’t realize) is that it also means that if someone with an AOL feedback loop “whitelist” thinks this gives him carte blanche to spam AOL users, then AOL has all his IP space, and no prizes for guessing how fast all that space can be added to AOL’s filters.

About CSV - I am not reallly going to make it mandatory.  It is an interesting idea and a fairly good one if it comes into its own.

We’ve been doing an adhoc implementation of csv for over two years now - in fact, when I first met Dave Crocker in early 2004, and I told him about it, I jokingly called our implementation “HPF” or “HELO permitted from” - where, for example, we’d look for HELO yahoo.com from non yahoo hosts [hosts with non yahoo reverse DNS], and treat that as spam sign.  A few discussions with Dave later (at other conferences that we met at), and over the next few months, I gradually got interested enough to dig a bit more into CSV.  It does look like a workable idea.

But, there’s no usable code to plug into most MTAs for csv checks, and it is still a quite early draft. 

We’ll also have to use the lessons learnt from deploying other authentication protocols to visualize potential problems with CSV and see that history doesnt necessarily have to repeat itself.

I do know that Dave Crocker, at least, is quite aware that edge cases can’t be treated as cavalierly as some people think, and that given the widespread use of email, any edge case can be very thick, accounting for lots of users.

So he and the rest of the CSV authors will definitely work to weed edge cases out before letting CSV out into the wild, and quite rightly too, production mail systems are not really a good place for experimentation of this sort. But that means it’ll take quite some time before CSV is released in a usable form, with MTA plugins and all, and before its use would be encouraged (possibly integrated with domainkeys would be my best guess - the two seem to address different sides of the same issue, but complement each other quite well).

And that means that people eager to embrace a quick fix solution will rush to adopt spf, possibly with the bandaid offered by trusted-forwarder.org. That site is an ever growing list of IPs that are known to be forwarding servers that don’t do SES return path rewriting, and the spf website says something to the effect that as a purely temporary measure, sites that use spf checks to reject mail should strongly consider using trusted-forwarder.org to exempt email from known forwarders, from spf checks. 

Well, it is not the first time that something initially set up as a bandaid (and in this case, a bandaid to patch a bandaid) has become a quasi permanent feature of some spec :)

regards
—srs

Microsoft’s solution for the phishing email problem is advocating the adoption of Sender-ID, their proprietary use of SPF path registration records, which they claim verifies the sending domain. Based upon the domain selected by Sender-ID, Microsoft also intends to compile a reputation. Microsoft describes their Sender-ID process as “Sender Authentication.” Proper authentication is critically important, as abuse attributed to the sender will impact the delivery of their messages. Most would expect that Microsoft’s “authentication” process actually verifies which domain sent the message. At least, this is the claim made by Microsoft. Surprisingly, this is not the case. It would be like me saying the postal service is authorized to deliver my letters, and therefore any letter you receive from the postal service bearing my name, is authentically or genuinely from me.

It is common for many different domain owners to employ the same email provider, and for the domain owners to also employ more than one provider. The proposed Sender-ID is a method for a domain owner to indicate which email providers are authorize to send their messages. When this authorization list is declared complete or close-ended, email will go missing in many cases. However, Microsoft warns that declaring this authorization list open-ended may cause the domain’s reputation to suffer, as abusers would still be able to utilize their domain. The publishing of the Sender-ID authorization by a domain owner is public knowledge, which unfortunately may assist abusers wishing to make convincing messages that falsify the sending domain, but which are then marked as “authenticated.” Sender-ID may not prevent those spoofing the sender domain, it only changes how spoofing is done.

Contrary to claims made by Microsoft, Sender-ID does not authenticate the sender, it only authorizes the provider’s email servers. There is a world of difference between granting permissions to anonymous providers, and verifying which domain actually sent the message. So what magic wand did Microsoft use to transform “provider authorization” into “sender authentication?”

Microsoft’s next step is to include SmartScreen, where the “selected” domain owner receives a reputation based upon user feedback. Consumer safety and protection is ensured only when assurances and reputation are based upon verified identifiers that directly act in the sending of the message. Sadly, the Sender-ID “selected” domain owner may have had no part in sending the message. To prevent innocent domain owners being condemned as spammers, or recipients being duped by false assurances, Microsoft could have based their assessments upon the name of a key validating a signature within the message, as with DomainKeys, or the name of the email provider’s server. Either of these alternative identifiers could be authenticated as acting to send the message. Reliance on these alternative authenticated identifiers, such as DomainKey, or server name, would not alter how email is handled or cause valid messages to be lost, unlike the problems created by Sender-ID.

Sender-ID constrains the domain owner’s choice of providers and will likely create endless consumer complaints. Importantly, Sender-ID leaves email providers unscathed when they permit a security breach. Security lapses may include open relay or open proxy, where a domain owner’s reputation may become seriously harmed. Sender-ID forces the consumer to trust providers, and provides no means to verify the actual source of a problem. In addition there is no consensus which mailbox-address must be protected, and which mailbox-address a reputation assessment is based. Sender-ID does not allow domain owners to publish SPF records, even if only for white-listing purposes, and be assured not to be included in Microsoft’s insidious reputation scheme. The reputation scheme proffered by Microsoft is their tool to punish domain’s who do not use providers that license Sender-ID.

we just have to wait for all the false positives sudden and widespread deployment of this will inevitably cause

sitting back and saying “i told you so” can be fun, but the consequences of doing so in this case are beginning to scare me.. hence my suddenly becoming vocal and outspoken on something that I have been considering for over a year

regards
—srs

SRS,

You are welcome viz the IETF history and I appreciate your update about CSV.

Just to expand a bit on your comments:

* Any form of email authentication will require behavioural changes.

* There are two parts to the exercise, “mail stream” and “message” authentication.

* CSV’s focus is “mail stream” authentication using the domain in the EHELO/HELO command and the IP address, so validating the mail server. It also creates a platform for integration with reputation/accreditation systems.

* Unfortunately, many mail servers are not “properly” configured to generate the correct domain in the EHELO/HELO command.

This means people will have to ensure their mail server configuration is RFC compliant.

* The application of email authentication for the SOHO is problematic.

Sadly, when the topics of authentication, reputation and accreditation are discussed, this market arena is often overlooked.

This is unfortunate. The micro business owner’s ability to utilize the Internet as a vehicle for empowerment is a powerful economic engine.

In the SOHO market place, domain owners often use shared servers for hosting and mailing. Individuals typically use their residential IP address to connect to the Internet and then log in to the shared server to do their mailings, or work on their web site (s).

Many hosting companies do not require smpt authorized log-in to send mail. Often individual domain owners do not have a dedicated IP address, but rather use a shared IP address.

If they have access to a dedicated IP address it may lack a valid reverse DNS entry. The shared server mailing software is not able to identify the domain sending the mail.

Even when the hosting company provides the needed basics so that the domain owner is able to point his or her domain to a dedicated IP address with a valid reverse DNS entry on a shared server, the default settings in the current version of Exim (as an example) do not properly identify the originating domain in the message headers.

* As to Doug’s comments concerning SPF version 2 pra scope, I suggest if you were to press the Microsoft design team, they would acknowledge this implementation is a “band aid” to respond to corporate concerns about phishing attacks.

I understand Microsoft is working closely with CISCO towards integrating the Identified Internet Mail protocol. Who knows, maybe CISCO can persuade Microsoft to adopt their patent license.

* If one uses the wizard that Microsoft has commissioned to aid someone in publishing an SPF version 1/2 compliant policy record, this wizard will generate a policy record that ends in ~all. If the policy record is only useful for checking SMTP mail from, the wizard also provides the user with a policy record to deal with the pra scope that ends in ?all.

* AOL’s decision to use SPF version 1/2 records to simply verify the sender’s mail server(s) when checking mail streams against their internal white and black lists is an example of a safe and responsible way for a network to apply these records, while avoiding the risk of significant false positive levels.

John

John Glube
Toronto, Canada

About Cisco IIM, it is quite similar to domainkeys (e&oe devils in the details) that there’s at least some work being done on merging the two specs

CSV, based on IP address and HELO tends to make for a easier variety of validation (I wont say reputation here) than trying to validate based on the sender. It also rather neatly gets around the traveling email user issue, where people can send email from wherever (kiosks, blackberry devices etc) without having to restrict themselves to their ISP smarthost.

> This means people will have to ensure
> their mail server configuration is RFC
> compliant.

Am I alone in considering this a feature as opposed to a bug?

WRT smtp auth etc - I’d refer you to the ASTA guidelines, which appear to be largely getting integrated into the MAAWG bcp and code of conduct [still in a draft stage, but looks quite good].  So there is substantial weight behind these, besides their being quite proposal agnostc. 

BTW, SMTP authentication with a username and password to enforce relay controls while supporting roaming users is often being conflated with sender authentication - not necessarily a good thing.

Two points:

* One key reason I prefer CSV is the underlying design philosophy stated in the draft charter - “The techniques will be compatible with usage and operational practices for Internet mail based on applicable standards and BCPs.” I agree that requiring people to “buckle up” on their mailing practices is a feature, not a bug.

* As to the SOHO market, most micro business owners simply want to run a good, clean business. Like other business sectors, to deal with online abuse, work is needed to ensure mailing practices are “up to snuff.”

Hopefully incorporating the ASTA guidelines into MAAWG’s best current practices will aid this process. My only point is that people need to take care any best practices can scale down as well as up the “economic food chain.”

Kind Regards,

John

John Glube
Toronto, Canada

John,

Addressing the phishing problem has been a principle justification used by Microsoft for Sender-ID. However, closed-ended lists, which are intended to reject unauthorized servers by either SPF or Sender-ID checks, also cause some emails to go missing. Most financial institutions place a high premium on having their messages delivered, so this strict mode is just not practical. SPF, in lieu of Sender-ID, will cause greater losses when these path registration records are close-ended. It is not hard to see how Microsoft is advantaged, when applying reputations against mailbox-domains.

Sender-ID will utilize any version of SPF records to apply their proprietary algorithm. Domain owners wishing to ensure delivery, or totally “opt-out” of Sender-ID, while still providing SPF records, are instructed to publish two record types. The record scoped for Sender-ID must then utilize a neutral open-ended list. Of course, both records would need to be open-ended to ensure delivery.

While this may allow messages to pass either Sender-ID or SPF checks, there is then the Machiavellian Microsoft SmartScreen reputation process. Domains may remain lucky and this recommendation for “opting out” of Sender-ID provides the desired results. Once those domains “opting-out” become subjected to forgery attempts and “user feedback,” then these same domain owners may find their messages placed into “junk” folders or rejected rather than being accepted. Microsoft’s reputation process will punish those domain owners who do not comply with Sender-ID and use closed-ended lists. Microsoft offers “future” warning of this in their presentations.

Sender-ID’s phishing solution is highly flawed. Sender-ID will not always base acceptance upon the visible “From” address. Microsoft mail clients displays the “pretty” name rather than the mailbox address anyway. This behavior makes recipients vulnerable to forgery, even with a validated From mailbox address , as possible with S/MIME signatures. Reliable delivery requires open-ended lists, which effectively bypasses forgery protection. A non-strict list could be used to “weigh” the message within a filter. However, filters, to be effective, evolve into being a reputation system based upon weak identifiers.

A reputation system, to be fair, must be based upon authenticated identifiers. However, without signatures, no mailbox address can be authenticated. Sender-ID requires domain owners be at the mercy of the email provider. These providers not only must be diligently securing access, but must also guard which mailbox-domains are used by their clients, with respect to both the bounce address and Microsoft’s Purported Responsible Address.

When a provider for either the sender or the recipient allows inappropriate access, the domain owner’s reputation may become seriously harmed. Sender-ID assumes universal compliance and security, but this is simply not reality. Much of the abuse is introduced when taking advantage of a lack of security. Authentication of identifiers permits locating security breaches and also protects consumers from having their reputation unfairly tarnished. These domains often serve as a trademark, but may become unusable due to Sender-ID’s lack of mailbox-domain authentication. This problem remains well beyond the control of the domain owner, unless SPF records are not published.

I’d like to challenge John’s two criticisms of CSV:
1)It doesn’t take into account SOHO users.
2)A significant fraction of mail servers don’t HELO/EHLO properly.
aren’t valid.

Consider one user of elvey.com, my dad.
He mainly sends email from 4 computers: his desktop at work, two at home (the one he usually uses and the one he uses when that one isn’t working, which is not infrequent) and his blackberry.  He also uses webmail from time to time.  He usually wants to send email using his elvey.com address from all these systems, but sometimes he sends email using other addresses.  For example, he is having trouble, and calls his broadband isp (roadrunner) for support, or calls his office for support, and they walk him through changing everything to use an @rr address and perhaps smtp server, or a work address and smtp server.
With SPF, this is a nightmare scenario.  Use of each address must be through an SMTP server that is authorized for that address!
That’s dozens of smtp server settings to configure - I’m sure that most of them are ‘wrong’.  THAT’s not SOHO friendly!
Now, consider CSV.  Dad doesn’t have do do a thing.  Everything just works.  If it didn’t, it’s almost certain that many people have already called the ISP or whomever and they’ve set up their SMTP server to HELO correctly.  In fact, it’s almost certain that it’s already set up correctly today, as mail to AOL and many other destinations has been bouncing when sent from servers that aren’t set up correctly. Sure, because there are so many mail servers, there are “many” that aren’t set up to HELO correctly, but as a fraction of the sources of legitimate mail traffic, it’s small.  The only issue is that if the ISPs fail to keep their noses clean, they’ll find their mail queues balloon, due to greylisting.  Due to ballooning support costs they’ll be motivated to clean up their act.  The SMTP servers that the SOHOs use have already set up their server to HELO properly, and that was the only thing they needed to do, if their noses are clean.  If not, they’ll need to set up CSV records that point to a reputable reputation service that will vouch for them.  The SOHO user just needs to have a provider that isn’t in the small minority of providers that don’t HELO properly, which they almost certainly already do; if they don’t they already have a major deliverability problem, e.g. all their mail to AOL is bouncing.

I’ve presented this argument on ASRG before, but not in the same way.  Perhaps using my dad as an example makes it clearer. (Comments via email appreciated.)  Clearly I didn’t get through to Newton…

As I see it, CSV is just a very low-cost way to assist current techniques to significantly reduce false positive and false negative rates.  Thus the arguably thick edge case of invalid HELOs isn’t a critical problem.

As I see it, Sender ID is just a very high-cost way to assist current techniques to significantly reduce false positive and false negative rates.

Payola.org,

I appreciate your comments. Perhaps I did not make my point clearly. Email authentication will require change. With CSV the objective is to ensure compliance with the technical protocols and best current practice guidelines for Internet mail. I believe this is a feature.

My rationale for raising the “micro business owner” was to simply remind everyone involved with email authentication and reputation/accreditation that this effort has a fundamental bearing on all business sectors and must “fairly” scale.

John

John Glube
Toronto, Canada

Matthew,

I regret the error in addressing my comment to “Payola.org” instead of Matthew.

John

Doug,

Please excuse my delay in response. I appreciate your taking the time to express your concerns about open records.

I wanted to do a bit of a digging.

* A large number of SPF records (version 1) for ‘legitimate’ commercial operations have been published.

* I don’t have accurate figures, but going through the various public forums, until late last fall, the general consensus was to publish a closed record.

* Unfortunately, anecdotal evidence suggests many folks who published closed records did not fully appreciate the related difficulties.

* At this juncture, to avoid disruption to email delivery, I would suggest that networks and third party filtering houses be encouraged to only use SPF version 1/2 records to verify a bulk sender’s outbound mail server (s) and then only use this information when checking the mail stream data against internal or external white, safe or black lists.

* I appreciate that MSN/Hotmail’s goal is to use their pra scope to authenticate mail and then use this result to generate an authentication indicator for the mail user agent.

Unfortunately, the platform chosen by MSN/Hotmail to initially achieve this objective, being SPF version 1/2, has various “edge” cases.

Also, the SPF Council which has taken over responsibility for SPF version 1 is not prepared to work with MSN/Hotmail. Depending on your perspective this may be good or bad.

* I am not sure how the permission based sending community will respond.

These folks are simply interested in getting their requested mail delivered.

They don’t really care about the ongoing squables between the various design groups. In essence their stance is very pragmatic:

“Tell us what to do and we will do it. But, if it does not work, or there is serious risk of significant false positives levels, don’t ask us to be your test bed.”

The public statement by SRS that Outblaze.com has removed its SPF records because SPF has failed to deal with the “edge cases,” which pose a serious risk of significant false positive levels is therefore very important.

I view SPF version 1/2 at best as a band aid and not a permanent solution. But, if the band aid is going to result in more serious problems, then the large consumer networks need to exercise responsible leadership to avoid causing deleterious harm to the email infrastructure.

Yes, I understand that Andrew Newton’s draft document Considerations for the use of the Sender Policy Framework was met with dismay by some in the SPF community.

Unfortunately, every time someone tries to raise issues, especially when based on hard facts, one is met with cries of “FUD,” “your agenda is to help CSV,” “you are helping spammers,” or worse.

The concepts of email authentication, along with accreditation and reputation can bring benefits and help build a suistainable model for email.

If a proposal has flaws or requires such a fundamental behavioural change that implementation costs out weigh the benefits, confront them. As appropriate, change the design. That is why its called an experiment.

(On this point, the reader may wish to read Are You Ready To Make Email Work For You?)

But, I am not comfortable with technological efforts which take on the air of a religious movement. Real world feedback has to play an extremely important role in the design and implementation of any scheme (s) to implement these concepts.

Presuming that the CSV design team is ready to release CSV into the “wild,” it would be my suggestion that:

* One or more of the large consumer networks encourage senders to publish a CSV record for client server verification. This will serve two purposes. It will require senders to ensure there EHELO/HELO commands are in order. It will also set up a reliable platform for use with reputation/accreditation services.

* The CSV design team and the large consumer networks work together in designing a “plug-in” based on the CSV platform that can work with the large consumer network’s proprietary message authentication schemes for end users.

* The large consumer networks and others continue their ongoing work in implementing a light weight cryptographic solution for message authentication based on the design work done by Yahoo! and Cisco.

John

John Glube
Toronto, Canada

John,

While CSV offers a means to defend network resources and provide a safe basis for applying reputation feedback to combat phishing, the immediate desire is focused upon increasing the integrity of the email sender. This need is not well met by Sender-ID, but can be much better protected with DomainKeys or S/MIME. Neither of these signature schemes, nor Sender-ID for that matter, offer resource protections. CSV will likely become a follow-on methology to fortify signatures and provide email policy assertions. A name based reputation scheme can be defended using CSV, as the initial step in the process, but this does not need to be the first step implemented.

People need to understand the risks, when their email provider suggests that they publish SPF/Sender-ID records for their domain, which have the effect of authorizing servers. They likely will hear the canard “SPF records will prevent your domain from being forged.” This claim is shamefully false, and publishing these records is not without serious risk. Not likely mentioned is that this mailbox-domain authorization scheme is to be used as a basis for compiling reputations, which is a wholly unfair and unsafe method for identifying abusers. Providers may hope this reputation scheme will supplant their monitoring efforts, which would have otherwise abated abuse at the source. As a result of publishing these records, domain owners may have their email blocked, when forgery occurs despite publishing these records. This is especially true when records are open-ended to ensure email delivery.

There are corrections and additional information that needs to be added to the SPF consideration draft. A broader view of this approach may clarify the superiority of alternatives, such as DomainKeys.

John - interesting comments there. SPF’s worst enemy is the fervent advocacy that’s driving it, right now.

So, Doug, what would you say to a combination of CSV and domainkeys, to kind of mutually complement each other?  How feasible is it, and where would you say the gotchas are?

Suresh,

There are two issues concerning DomainKeys which could be called “gotchas.” One would be the “replay” sure to happen within larger domains. The other would likely occur when marketing decides they can issue keys to millions of individual users.

Adding a “revocation identifier” within the signature header would permit a convention where every domain publishes their own black-hole list as a method to defeat “replay” attacks. Monitoring queries to this list could alert administrators to a possible replay problem as it occurs. Here CSV would play a useful role for conserving resources. This would establish an authenticated name safely used to establish reputations. If this name is within the domain of the signature, there would be no need to make additional reputation or revocation checks. CSV could thereby offer a significant reduction in overhead. Already CSV ensures only a single lookup is required, where this efficiency can be extended to also protect the signature process.

The CSV record provides an additional 15 bits that can be used for domain assertions. A bit may declare all messages are signed by the “From” domain. Currently, one bit is used to declare all outbound SMTP servers offer a CSV record.

This is explained more fully in:
http://www.ietf.org/internet-drafts/draft-otis-mass-reputation-01.txt

The issue of improper key deployment could be handled by removing the “g=localpart” optional extension for a key. This would acknowledge that assurances and reputation only resolve to the domain, and not the localpart of the mailbox address. Adding this type of option is a slippery slope. The next option may include limits on links found within the message, or whether the key can be used within the Sender header, etc. DomainKeys could be kept simple by removing the “g=localpart” which would have the added benefit of discouraging per-user keys.

The concept behind this “g=localpart” feature was for issuing private keys to third-parties so they could sign messages within the same domain, but limit which localpart the key permits. The “Sender” header was to fulfill the role of permitting users freedom with respect to the From header. Restricting the localpart within the Sender header will not have the desired affect sought by the “g=localpart” option. Allowing this to use the same domain will not afford recipients much in the way of knowing who is the controlling authority.

Sender: .(JavaScript must be enabled to view this email address)
From: .(JavaScript must be enabled to view this email address)

Domains with millions of users, which expect this “g=localpart” key restriction therefore permits users to sign their own messages, could easily become abusive to DNS.

Assuming that people pay closer attention to the domain, then by not having this “g=localpart” option would likely force separate sub-domains in the few cases where it might be needed.

You may receive a message:

From: .(JavaScript must be enabled to view this email address)

It would make it clear the content of the message (including the localpart) is governed by third-party.big-bank.com and NOT big-bank.com. Campuses could use a descriptive sub-domain that indicates the localpart is not checked. Even a flag for this purpose may be helpful.

From: .(JavaScript must be enabled to view this email address)

Once messages become signed, few institutions will likely wish to mix domains they control using outbound filters, with domains they can not control. Having private key delegation remain in a separate sub-domain seems the far safer alternative. It would also protect DNS from likely egregious misuse.

ISPs already have the ability to force compliance to anti-spam policies. . .

1. Force authentication - if everyone who wants to send e-mail must send a username and password to send, that substantially cuts our all non-authorized e-mail. If a machine is infected, the account can be disabled as soon as the rough operating system is noted by the service provider. No free rides for anyone.

2. Use SPF1 with “-all” on all domains. We do this quite successfully. No one can send mail via our servers unless someone they authenticate and then our SPF records validate the fact that the message actually came from the server. We have an SPF record created for every one of our customers and only mail that originates from our mail servers passes the SPF tests.

3. Make anyone sending e-mail to use IDENTICAL information in the E-MAIL ADDRESS and REPLY TO ADDRESS of all outgoing messages. The information contained within those fields must exactly match the e-mail address under which the message is being sent. If your customer"s e-mail client is set up any different, refuse to transport the message - no exceptions.

4. Enable port 587, as defined by RFC 2476 for the use of roaming users.

5. Mandate authentication on port 587. No free rides for anyone.

6. If a user violates the policy, disconnect them.

7. If another ISP violates the anti-spam policy, block all incoming messages from them.

8. Block all incoming messages from any domain where the mail server does not have a PTR (reverse DNS) record. Refuse them before they are ever received. AOL already does this and any legitimate mail server now has a PTR record that can be validated.

9. Check all mail servers for the acceptance of DOMAN LITERALS, ie: user@[XX.XX.XXX.XXX]. This is already mandated by RFC2821 4.5.1. We do this and quarantine all messages that will not accept queries for the domain literals. If we find a domain that does not comply, we refer them to www.dnsreports.com and have them run a report on their name. It sometimes requires some additional explanation on our part, but many of them not only comply, but begin to check for acceptance of domain literals as well.

10. Block all incoming messages from domains listed in the SPAM DATABASES. We run a check on all incoming messages against SPAMCOP. If a domain is listed, the message is dumped. SPAMCOP only lists for 48 hours and then rescinds the listing. If the sender"s domain is listed frequently enough and they ignore the messages they receive from SPAMCOP, the don"t deserve to be sending e-mail and adding to the bandwidth on the internet.

11. A suitable period of time after SPF is accepted and becomes part of the RFC, refuse to accept messages from any domain that does not publish an SPF record using “ALL.

Yes, there will be some yelling, screaming and stomping of feet in the beginning, but, eventually, the dust will settle and we"ll all be able to get back to the task of running our businesses instead of allowing spam to raise our blood pressure!

The problem is not controlling spam. The problem is getting ISPs to quit being afraid of their customers and take a stand.

SMTP auth - fine. Port 587 - excellent.

From and reply-to should be the same - whatever for?  And why?  I frankly cant see your logic here, or the utility of such a measure.

SPF -all enforcement: your network, your rules. if you have a userbase that is more than family + friends, you would want to reconsider what you said, or wait for lots and lots of user complaints about lost mail to roll in, which just might help you change your mind.

Bruce,

Demanding that your customers publish SPF records which authorize your servers, should obligate you to ensure they are protected, without also demanding these records be “closed-ended.” Reputation schemes based upon the “authorizing mailbox-domain” (often erroneously called the “authenticated mailbox-domain”) is selected by either the PRA algorithm, or the bounce-address, when any SPF record has been published. For example, as with the proposed reputation extensions to Outlook, the PRA algorithm may be used against any SPF record type.

Providers must ensure protection by overlaying message headers and bounce addresses, or inspect both the PRA headers and the bounce address for any potential conflict with all other customers. The PRA check requires a license from Microsoft. Reliance upon either the bounce-address or PRA check will cause the loss of some emails, when publishing close-ended records. Leaving records open-ended is the only means to ensure delivery, however this will not afford your customers protection, without additional measures taken by you, the provider. Otherwise, any spoof allowed to enter your system will appear fully authorized.

What assurances will you offer customers exposed to a new reputation vulnerability, created by your request that they publish SPF records? Will you inform your customers of these risks incurred when publishing an SPF record? Will you obtain a license from Microsoft? What damage of their reputation will you be liable for, when tarnished by an open-proxy or open-relay permitted within your network? What liability will you accept for the PRA header or the bounce-address not being inspected? What mechanism will you provide your customers to determine the cause of their bad reputation? What logs will your customers be entitled to see during a discovery process? What does this exposure mean for your other customers?

Would it not be safer for your customers to use DomainKeys? Why are you not requesting your customers use the safest method available? Should you be liable for providing bad advice, without offering a full disclosure of the related risks? Don’t be naive in thinking that just because SPF/Sender-ID keeps you anonymous, you will not share in the resulting damage.

DomainKeys can maintain the integrity of email delivery, whereas SPF/Sender-ID is a serious threat for the majority of domain owners.

Doug et al;

We handle substantially more than just family members.  Our user base is extremely diverse and ranges from hospital surgery centers to e-commerce sites to conventions that are in Chicago only once a year but get e-mail and web traffic from all over the world.

We automatically publish the SPF records for all of the domains we host.  We run both internal and external DNS servers and have complete control over them.  None of our customers user mail servers located outside of our network.

If you will re-read my original comments, you will see that we don’t yet enforce “SPF -all” because SPF is not yet an RFC.  Once it becomes an RFC and a long enough period of time has elapsed, we WILL enforce “SPF -all”.  I suspect that many other ISPs will consider doing this as well.

Will this cause some lost messages?  Probably so.

Without someone being willing to take an extremely radical step, we will never get rid of the problem as we know it.

As of right now, no one is complaining about lost messages.  We are not yet enforcing “SPF -all”, but we are auto-deleting all SPF FAIL results.

With regard to the enforcing of “from and reply-to” not only being the same but exactly matching the actual e-mail address that is used to send the messages, this closes one additional loophole that might be used by someone who is a customer and spamming.  This also insures they will be in compliance with the SPF record published for their domain.

The one thing I failed to mention in my original post is that we also check HELO/EHELO records at the time the originating mail server connects to our server for mail delivery.

We have already reduced our spam by more than 95% - without complaints of non-received e-mail.

Bruce

PS:

SPF does NOT require a license from Microsoft.

Sure, it certainly doesnt require a license from Microsoft.  But I must remind you that even if it does get an RFC assigned to it, there’s no MUST requirement to use SPF, as opposed to (say) domainkeys, or CSV.

Right now, I would put it to you that the technical (de)merits of several proposals have taken a backseat, and market forces (such as the userbase each of these proposals manages to gather) will determine the success or failure of these. 

And in the long term, after the effects of the marketing has faded out and the various proposals out there (domainkeys + IIM, some variant of spf + microsoft’s sender ID, csv and batv in conjunction with domainkeys would be my guess), the proposal[s] that suck the least will gradually start getting more users.

Suresh;

We can debate this all day long.  This will not change either my position or the fact that we have already chosen to adopt SPF, along with other control measures.

With regard to Microsoft, we will NEVER purchase a license to use an anti-spam product they might developed.  We already have the ability to run free software on unix systems, why would we want to pay a company that has set a new standard in corrupt vaporware for a product that they basically stole from another software author?

If there is a major shift in the industry, we will consider looking at the tools that are used by other ISPs.  For now, I stand by what I have written above and implemented for our users and customers - they are happy.

Stop your feat, yell and scream, give me and the developers of SPF as many ?technical (de)merits? as you like, it will not change my position.

a. I’m not suggesting you run anything at all that’s not licensed under a FLOSS license [free/libre and open source software, a term coined by an old friend, Rishab Aiyer Ghosh, btw -  http://en.wikipedia.org/wiki/FLOSS)

b. It is a free country and you are welcome to use whatever you wish. Expecting that use of the proposal / technology of your choice is going to become mandatory is rather wishful thinking.

Here are some questions that I have.

Mr. Newton edited a draft document which to my understanding is based on field testing done by MAAWG which reads:

Considerations for the use of the Sender Policy Framework

This document sets out some very legitimate issues when using the Sender Policy Framework.

What is the response? A variety of forms, but they seem to boil down to:

* the document is flawed;

* you can’t trust the editor; and,

* ISPs need to stop worrying about the whining of their customers.

I have yet to see a response which says, these are serious issues. Should we re-think the design?

Maybe the value of SPF version 1 and 2 is limited to identifying the outward bound mail server of the sender for mail stream authentication.

If so, is there a simpler way to achieve mail stream authentication that is less demanding on the DNS infrastructure (aka validating the mail server) that better supports network security and a better way to achieve message authentication through the use of a light weight cryptographic proposal (aka authenticating the sending domain)?

Don’t take this the wrong way. Those networks that were early adopters of SPF have greatly helped the online community in sorting through what works and what does not work.

But, now that the large consumer networks have run tests and pointed out concerns, we need to say, “Okay - this was a useful exercise.”

Let’s move to the next stage. As to SPF version 1 and 2 becoming an RFC, it presently is on the experimental track.

As was pointed out to me, there is a significant difference between a proposal that is on the experimental and standards track.

The underlying point we need to focus on is that email authentication will not stop UBE. Yes, with the correct approach, it may help to stop some UBE senders from hiding their identity.

However, we have bandwidth providers willing to sell bandwidth to entities which send or support the sending of UCE.

Sending UCE is not illegal per se in the United States, providing the sender complies with some basic guidelines and can find a carrier willing to transmit the UCE.

Given this stance and the desire of carriers to make money from those sending UCE and those wanting to stop UCE, realistically the minimum standard is that:

* US bandwidth providers take reasonable steps to ensure that all commercial and transactional or relationship email sent over their networks is compliant with the legislative requirements.

* US bandwidth providers that fail to take reasonable steps to ensure their networks are transmitting compliant commercial or transactional email or are used to sell software which will aid people in sending non-compliant commercial and transactional or relationship email are jointly and severally liable for violating the law.

Would it be wonderful if US bandwidth providers said no to UCE and really meant what they said? Yes.

Until that happens, we will continue to see the ongoing struggle between bandwidth providers that cater to UCE and those within the community opposed to all forms of UBE.

What about the huge number of internet access services that have to respond to the resulting deluge of compliant and non-compliant unsolicited bulk email?

Also, since many UBE senders don’t wish to pay for bandwidth, another difficulty is how to deal with the huge networks of compromised personal computers?

Would having the whole world adopt email authentication resolve these problems? I will venture a view. Not really and I suggest these are the underlying difficulties we face.

Having said this, developing and implementing the right proposals for email authentication which can form an underpinning for scalable and fair reputation and accreditation schemes can aid all senders in getting their mail delivered.

John Glube
Toronto, Canada

To throw some fuel in the fire, right before I shut my laptop and board my plane .. here’s Miles Libbey from Yahoo on sender authentication.

https://antiphishing.kavi.com/events/Conference_Notes/sender_auth_myths_and_domainkeys.pdf

Bruce,

You are publishing SPF records for your customers domain, without ensuring the PRA header is not in conflict with other customers. However, Microsoft has announced they will compile reputations based upon these SPF records against the domain selected using their PRA algorithm. The PRA header is not the bounce-address, the Reply-To, or the From header, but instead is dependent upon their proprietary algorithm where the From header has the lowest priority and does not consider the Reply-To header at all. A lack of defending against such spoofing, places you and your customers at risk, imposed by Microsoft’s ubiquitous email clients.

A misconfigured firewall, or MTA within your networks may also cause your customers to suffer harm to their reputation, specifically due to these SPF records that you published “supposedly” on their behalf. Microsoft does not intend to wait for RFCs to utilize these SPF records. They may even view their use as a means to influence a need to license their algorithm. Many of Microsoft’s protocols fail to become an RFC. There is no means to exclude you and your customers from being assessed by Microsoft’s unfair reputation scheme, once SPF records are published.

There is no means for a sender to know which recipient forwards their email. SPF will cause the loss of valid emails in many scenarios, where solutions to resolve these problems through anomalous handling is different from that required for Sender-ID. Why advocate wrenching changes, when there are better solutions that do not attempt unscalable path registration? The environment you describe is not universal. By publishing open-ended records, and applying path registration rules for the bounce address, or even the bounce-address, Reply-To, and From , will not ensure your customers receive adequate protection. While you may think this “radical” step will get rid of a problem, it will actually make the problem more damaging. You are exposing domain owners to reputation risks well beyond their control, when spoofing continues to plague them.

Perhaps abusers take advantage of your desire not to license, or your customers desire to assure delivery of their emails. These spoofing problems become more intractable with the lack of consensus which mailbox-domain’s path is being registered. The possible complex paths a message may use, makes path registration difficult, if not impossible, to defend.

Your use of screening incoming HELO/EHLO server is commendable. It should be practical to perform this step using CSV, where only a single lookup is required, rather than the 102 lookups required by SPF. The use of CSV for this purpose will not expose your customers to reputation risks, nor will this require a license incompatible with open-source software.

With a majority of domains not registering their email paths with SPF, success at quelling 95% of spam must be based upon some other mechanism. Spammers also publish SPF records as well. Would this success be due to the HELO/EHELO? Responsible email providers should avoid placing their customers as risk, and instead employ DomainKeys and CSV to authenticate the source of email.

If there are problems related to sender reputation with DomainKeys, only those logs for messages signed with the specific domain’s private key would need to be inspected. DomainKeys would allow signatures to remain within the control of the domain owner, and not require complete trust of the email provider as well. When there is a problem created by Sender-ID, a discovery process will need to examine logs that include all your customer’s messages. Sender-ID/SPF is hardly a practical scheme to ensure fair assessment of reputations, nor can Sender-ID/SPF offer an assurance of privacy.

John said:

“I have yet to see a response which says, these are serious issues.”
Response from whom? I certainly said this on spf-discuss:
http://www.gossamer-threads.com/lists/spf/discuss/19008?do=post_view_threaded#19008
the draft has gross flaws.
I wonder how “Bruce” would handle the “nightmare scenario” I presented in a previous comment, above.
How many users typically send with a From domain with a -all?  Who supports them?  What is the support cost?  These are vital questions for MAWWG that remain unanswered.