CircleID

But meng, how or why do you posit that email is moving towards default deny?

Some dnsbls blocking everything they can see?  Nobody with a lick of sense uses them on a truly production mail system.

Or is it cisco's cutesy ads that advertise their firewalls / DoS mitigation appliances etc?

Or, as has been attracting a lot of recent circleid attention, is it that Goodmail is being used to attempt to make senders of solicited transactional and marketing email share some of the costs of that email with the ISP?

That concept is kind of a laudable idea, by some lights - but i have yet to see how well it will scale .. at least goodmail is limiting its scope and aggregating payments, unlike most of the harebrained online stamp and epayment schemes I've seen…

One case study both AOL and goodmail may want to look at is a korean ISP called Daum, that implemented a system that was basically "buy stamps from us so we'll whitelist your bulk mail. We'll do this only to commercial senders, and we'll let our users vote on how solicited / unsolicited your email is when its stamped...".  I have not seen or heard of it recently thouth at least with senders who concentrate on the korean market, it was a very significant thing, because Daum is to korea what AOL is to USA internet users .. the 800 lb gorilla

This presentation by Jaewoong Lee, CEO of Daum, describes their system. Some parallels can possibly be drawn to the goodmail approach.  Yes I know there are quite a few differences, but this is the closest thing to a case study / prior history of this happening that I can think of.

href="http://www.apcauce.org/meeting... [PDF]

Meng's article suggests a dichotomy: that the world will be "default accept" or "default deny", and the latter choice is the only one that will solve the problem of phishing. This is not a sound argument because the dichotomy is false. While "default accept" and "default deny" are indeed mutually exclusive, they can be applied to a much finer grain than the world as a whole. We can ask of individual email addresses whether the default rule is "accept" or "deny", and an individual can have addresses of both kinds, reserving the "default deny" address for important correspondence.

Furthermore, the emphasis on the default rule fosters a simplistic view of things by ignoring rules other than the default. A stack of rules could first start with a whitelist (offering preferential delivery to the main inbox), then go through a series of blacklists, and end in "default accept" (offering delivery to a secondary inbox). An email address governed by such a rule stack would technically be "default accept", but to think of it in those terms would be to miss all the important aspects.

Anyone who is looking for further reading material on this subject may find my CEAS 2004 paper, Beyond Identity [PDF], a worthwhile read.

Both whitelisting and blacklisting have the same flaw at the moment.
They both have a problem establishing the identity of the sender. 
Whitelisting an e-mail address is all well and good, until a friend of yours has his machine taken over, has all the email addresses harvested from his address, and then used as a spam sending zombie.
If the SPAM were to come at you using e-mail addresses from people in your white list, then it would land straight in your inbox.
The reverse of blacklisting using e-mail addresses is already laughable due how easy it is to spoof e-mail addresses, even though that’s all what many webmail accounts and applications offer in the way of blocking.

Doing it on IP address faces similar problem, you can't really whitelist an IP address, as most people aren't on static ones.  And blacklisting an IP address will cause legitimate mail to be blocked as the spammers may not be the only one that are sending from that IP address.
Those sending the spam have long ago learnt to keep mobile and so you are always one step behind.

The fundamental problem that has to be solved first is to replace SMTP with A-SMTP or similar, so that to send mail you first have to authenticate yourself on the mail server, and it then compares the e-mail address you are using against a list of valid e-mail addresses for that account.
It also allows you to set limits per account of how many e-mail you are allowed to send.
Secondly the ISP's have to block all those zombies on their networks by blocking or redirecting port 25.

At that point whitelisting/blacklisting mail servers will become feasible.

Meng, I think you hit the nail on the head here in your usual eloquent way.

Suresh, the reason we all must move toward default-deny is that the overhead and administrative costs involved in our current whack-a-mole environment are unsustainable.  We need a system which empowers end users - to make each man an island.

This is bad news for the likes of us who make our livings whacking moles.  But it will be good for most end-users when they get to choose their corespondents.

It is of course a depressing development, as we move away from the original open design of the email system.  But it is better to move to a more closed system which is still basically fair and empowering to end users than to stick with the current one, with it's baroque balkanization.

-=Julian=-

The best way to stop phishing is for trustworthy entities to cryptographically sign their e-mails.  Unfortunately, few entities do this except security-notification outfits like US-CERT.

If a user receives an e-mail purporting to be from, eBay, but it's not properly signed and eBay has told their mail recipients to look for a valid signature, then the recipient should not believe it's from eBay.  Pretty simple.  All current mail clients, including most Webmail clients, support S/MIME.  So, why is e-mail signed so rarely?

I wish it were that easy.

Replacing smtp, or signing everything with s/mime - neither of these is the FUSSP (final ultimate solution to the spam problem) .. http://www.rhyolite.com/anti-spam/you-might-be.html

Some that seem particularly applicable -

knows-SMTP-4
You know that SMTP has no authentication and have never heard of SMTP-AUTH, SMTP-TLS, S/MIME, or PGP.

knows-SMTP-5
You know that the failure of SMTP servers to authenticate the SMTP clients of strangers is a major bug in SMTP instead of an expression of a primary design goal.

knows-SMTP-7
You have never heard of RFC 2554 or RFC 2487 and the FUSSP includes fixing the lack of authentication in SMTP.

programmer-8
The FUSSP involves certificates, but there is no barrier to spammers buying many independent certificates.

programmer-9
You know that certifying that a user legitimately claims a name and has never used some other name is cheap and easy.

programmer-11
The FUSSP involves replacing SMTP.

Nobody is claiming it is easy, and yes, just replacing SMTP with a better standard won't be the final solution to getting rid of SPAM.  In fact I believe that there will always be SPAM, and there is no final solution as long as there are people that want to send it. 
However if no action is taken, eventually the level of SPAM will be so high that email will die in an avalanche of SPAM, and nobody will want to use it anymore as it will be harder to send legitimate mail due to the levels of filtering and blackhole listings while your inbox still gets flooded by unwanted emails.
What we can do is make it harder to send SPAM, and at least throttle it a bit.

The best way to stop phishing is for trustworthy entities to cryptographically sign their e-mails.  Unfortunately, few entities do this except security-notification outfits like US-CERT.

Why not pass the cryptography to the SMTP layer? Signed emails in the lower level could easily be added by sharing the public keys on the DNS txt fields, thus providing a way to verify if the sender is allowed to send an email from a specific domain.

Following this idea, who controls the DNS controls who can send mail from the domain. Only using this approach one can deny-by-default all mail which claims to be coming from a domain but failed the signature test.

Meng wrote:

Can you imagine a Balkanization of messaging, where if
you want to talk to someone you have to first join their BBS?
I’m an idealist: I care deeply about the future of free
communications. I don’t want to screw this one up.

Too late Meng. We been doing this since the 80s. Logging into our system was NOT an option, never was, never will be.  The relaxation of authentications and authorization methods in the name of a 'open internet' caused major security problems.  We all knew it was all possible, but it was ignored and those who adhered to weak unsecured methods are now dealing with the consequences. It did cause major grief with the need to alter some secured designs to our online hosting (BBS) products, but we knew it was aberration and people will eventually come to their senses. This explains why we are have experienced a very high customer return rate - the promiscuity with an open internet was getting too dangerous.  It also explains the reborn direction is to have "login only" or membership systems.

Hector Santos, CTO
Santronics Software, Inc.
http://www.santronics.com

Nobody is claiming it is easy, and yes, just replacing SMTP with a better standard won’t be the final solution to getting rid of SPAM.  ...
However if no action is taken, eventually the level of SPAM

We need to be careful to distinguish activity from progress.  We also need to be careful to acknowledge the actions that are already underway.  In particular, the efforts to create a trust-overlay to email, where Good Actors, are vetted and their mail is subject to preference handling.

When we know exactly what functional changes are needed for email, and when we have tried to add them to the SMTP infrastructure, and when the attempt has failed, then we will need to consider replacing SMTP.

Until that time, any call for replacing SMTP needs to generate three questions:

1. With what?

2. Why will it be better?

3. What benefits of existing Internet mail will be lost?

Comments 5/8:

Ted - what you have suggested is called DomainKeys :-) Have a read about it - http://www.ietf.org/internet-drafts/draft-delany-domainkeys-base-03.txt.

I find that the problem is more taking a utopian view of this.

Short of keeping your computer off the net, ultimately, no technological fix can completely eradicate all phishing and spamming. There's always the chance that the bad guys will systematically attack systems across the internet, and defeat white-listing- indeed that has happened many times with various viruses. There's even a chance that your best buddy is a phisher without you knowing.

As I see it, the main problem is to contain/minimise the issue; so that the vast majority of the mail *sent* is not of this type-all mail on the internet goes through an ISP of some kind.

The bayesian mail filters seem to be handling the problem pretty well on the receive side of the equation- most of the bad emails get filtered out. That looks like it will continue to worsen the economics of both phishing and spamming to the point where the *flood* will subside to a trickle.

Still, right now, many ISPs are being irresponsible, and these ISPs need to be identified (using SPF) and made to apply bayesian filtering and other techniques to the send side- if the mail never enters the Internet, or made so it enters much more slowly, the load on servers will be slashed and the user experience will greatly improve worldwide.

That way we won't *have* to use white lists- white lists are expensive solutions involving every individual worldwide having to do work to update them. We need to apply more automated brute-force techniques to the problem.

a) There are lots of other solutions to phishing than default-deny, some of them quite promising.  Indeed simply adding warnings to, rather than blocking, mail from unknowns, including rewriting the URLs for warnings, would do a lot about phishing.

b) Though it's hard to get people to change sides on this, in the anti-spam community, there are those who view the content of the messages as the issue and those who think bulk mail abuse is the issue.  Long before we went to a world of default-deny, we would want to experiment with learning the difference between individually written and bulk mail, and applying any default-deny regimen to bulk mail.

Brad—email is a long long way from default deny.  And content is - mostly - not an issue (except for truly illegal content that's internationally recognized as criminal .. child porn and warez for example)

Unsolicited Bulk [and/or Commercial] Email seems to be a fairly good working definition, for all the hairsplitting that goes into the "let's first define what spam means" question.

Oh, and Ian—"Identify everybody using SPF"?  yeah, sure, when large ISPs are losing spf (http://www.circleid.com/posts/spf_loses_mindshare/).

DKIM does seem to have some amount of potential here, and the spec is not as fragmented as SPF currently is. At least it does tend not to bite people with .forwards the way spf does. But I wouldnt claim it is a cureall either.

Bayes?  Not very effective on a scale larger than your personal mailbox, at least when considered out of the realm of ivory tower research papers. And trivially easy to poison using random text "chomskybots" (the first bayes posioning random text that people saw in spam was a weirdly surreal jumbling of Noam Chomsky's writings .. which probably became more readable in the process)

You try more and more technical solutions in the vague hope they'll have any effect at all .. any lasting effect. Then let me know.

Congratulations, Meng. Whether I agree with you or not, it is always refreshing to read your attempts to frame the spam discussion in terms of a broader sociological context.

But how could you, a student of history who peppers his articles with obscure references that send the average reader to consult Wikipedia, use the term "final solution"?

The ashes of the murderer who coined the expression have been scattered in the Mediterranean; his boss died in a bunker.

Poor choice of words :-(

> Poor choice of words

Or a well chosen choice of words .. if Meng was out to troll circleid.

I'd say he's succeeded, brilliantly. And hooked lots of people.

I'm sorry if my choice of words offended you, Daniel—I wasn't trying to invoke Godwin's Law :)

FUSSP—the "final ultimate solution to the spam problem"—has become industry jargon, and "final solution" was short for that.

Besides, if you'll allow me this comment, the little man with the mustache has done enough harm—let's not allow him to damage our language too!

Hoping this doesn't earn me a $12m bounty, I remain

Your pal
meng

Meng,

You wrote:

Hoping this doesn’t earn me a $12m bounty

Not a $12M bounty but maybe a beer at MAAWG next week?

Yours,

Daniel

I've used a default deny email scheme, using the TMDA challenge response mechanism. It is an effective way to stop email spam, but almost whatever hurdle you place in peoples way will be too much for some legitimate correspondents.

In my case one of the things that stopped me using TMDA, was the failure of an intelligent, but busy (and let us not deny it, very attractive) lady to pass the challenge.

Also the concerns of my peers that such challenges, to forged addresses, might constitute spam. However this is an argument that is more about social agreement on what is acceptable, than technical issues, since well designed C/R systems mitigate the cost, by limiting the number and size of challenges. And the wide spread use of C/R would mean people would rarely see challenges to faked addresses (here there is sometimes a divide between mail admins who worry about load, and end users who care only about eyeball time; there are more end users than email admins).

As such if such a system gained social acceptance, it would be a workable way of reducing a lot of the unwanted email. But we can achieve as much, possibly more, with less intrusive systems.

However I hold that the primary source of the current email spam problem, is underlying security issues with widely used client software, and issues of monoculture and monopoly. Hence my article here about megaphones.

Phishing is a different question entirely. Whilst it might be possible to use anti-spam measures to restrict the effectiveness of bulk phishing runs, there will always be people trying to scam others, and such restrictions would only force them to be more selective in their targeting (stupid, rich people don't deserve to be scammed either).

As such the main reponse to phishing be legal, and not technical, although some simple technical measures may be worthwhile. The failure of governments to provide an effective response so far, has allowed the criminal element to flourish, but it is quite a small number of individuals, and easily solvable with political will. Governments have deployed transnational responses to other types of cross border crime, and I believe it is merely technical competence that holds many of the legal agencies back in this area.

I currently deploy greylisting, with blacklisting, and MIME type based rejection (for viruses), and trap about 99% of the unsolicited bulk email using these three techniques, with no end user interaction, and no false positives rotting is "suspect spam" folders, and very few reports of false positives or problems.

I strongly agree with the comments that content based filtering is of limited use in dealing with spam, and am utterly frustrated when I daily encounter false positives from various big email providers, who systems are both less effective, and more error prone.

The desire to change the world is very natural, but sometimes it is more effective to work out, and deploy effective local solutions. And share those with people who trust your judgement in these matters.

SPF stalled because it require global changes in how SMTP is deployed. The big providers generally advertising rules that said "our email servers, or maybe from anywhere else" which made it ineffective against backscatter, and it also failed to address the case where a similarly named domain is registered, and used with SPF, so limiting its effectiveness against forgery.

IP based blacklisting on the other hands stays fashionable, because despite its obvious limitations, it is simple and effective, and more importantly doesn't require a global change.

Isn't a final solution to phishing simply SPF?  In particular, if spam filters blocked any mail where the sending domain has an SPF record and the sending relay isn't on it, then any domain that wanted to prevent phishing could do so, simply by creating a correct SPF record?

Note, this is not a general solution to spam.  Just an observation that limited SPF is still a good protection against phishers.

So - citibank.com has a spf record. What stops phishers from registering c1t1b4nk.com and publishing an spf record for it?

"Reputation".

SPF's been touted as a whole lot of things so far. Unfortunately for it, as it turns out in the long term.

Now, "v=spf1 -all" is a handy way of saying that a domain sends no mail at all, one of the few features of spf that come in really very useful.

Spam filters (spamassasin+crm114) block almost all the phish I am sent.  Virtually all the phish that actually reaches my INBOX has a From address that is on my whitelist (e.g. service@paypal.com).  If e.g. paypal had spf records and I filtered mail from relays that were not present in those spf records, then basically all the phish I get would disappear.

No this system is not perfect.  In theory, some phish might be able to get through the spam filters as well, but phish mail actually has a lot of structure that makes them particularly easy for spam filters to catch.  e.g. you probably get very little normal mail that asks you to validate your account etc.

Er… i'm sorry to rain on your parade, but I got news for you.

PAYPAL HAS AN SPF RECORD.

Does that stop you from getting paypal phishes?

And what role do the rest of the non spf based rulesets in spamassassin etc play in stopping those phishes for you?

I only thought about this today because I finally read this article.  I have not yet personally configured by inbox accordingly.  Is there any reason to believe this plan shouldn't work?

As for recognizing phishes as spam, spamassasin usually misses the phishes, but crm114 is very good at catching them.  I assume that, in general, rule based filters will miss phishes, but bayesian-style pattern filters like crm114 are really good at catching them.  Note I have not tried the spamassasin bayesian stuff so I can't give a review of how it does against phish, but again I would bet it does substantially better than the rule based stuff.

Note: I have things configured so that if both spamassasin and crm114 agree that a mail is spam, then it is definitely spam.  If only one of them think it is spam, then it goes in a maybe folder.  Only if neither think the message is spam does it reach my inbox.  All mail that reaches my inbox is automatically whitelisted (unless otherwise designated).

The problem is not identifying genuine email from Paypal, SPF will help you do that.

The problem is stopping the other phishes, and where they use a domain they control, that record may also be approved by the corresponding SPF record the spammer inserts.

As such if you get phishes today, you'll get them if you use SPF, they'll just be from a slightly less plausible domain name.

The way that Paypal solves the problem is using certificates (similar to S/MIME) except the certificate is on their website, rather than on the email.

However you don't blame the postman if he brings you a physical scam letter, yet people blame SMTP for the virtual scams.

I must not have been clear before.  My point is that crm114 does a really good job at catching phishes in general. Mail with "plausible" but different from-addresses does not get through because those addresses are not on my whitelist.

The only phish that reaches my actual inbox has a whitelisted from addresses.  With SPF, that problem goes away.