What is DMARC?

This is very much a summary of DMARC in a nutshell (I will probably write an article about this in the future), but from the website:

A DMARC policy allows a sender to indicate that their emails are protected by SPF and/or DKIM, and tells a receiver what to do if neither of those authentication methods passes — such as junk or reject the message. DMARC removes guesswork from the receiver's handling of these failed messages, limiting or eliminating the user's exposure to potentially fraudulent & harmful messages. DMARC also provides a way for the email receiver to report back to the sender about messages that pass and/or fail DMARC evaluation.

When I first heard about DMARC, I said to myself "Self, why do we need another email authentication protocol?" The answer is that DMARC is not another protocol but instead leverages existing email authentication protocols and provides feedback to the spoofed domain.

SPF already provides a way to say: "If this message fails an SPF check, discard the message." It's called a Hard Fail. However, not all hard fails are illegitimate (there are significant false positives with SPF). DKIM, in itself, doesn't provide a way to discard a message if it fails an authentication check. This makes it less useful in securing the Internet (i.e., it is a barrier to adoption).

Besides which, what happens if an SPF check asses but a DKIM check doesn't? And if one of them fails, who should you tell? DMARC provides a mechanism that says: "If one of these checks fails, discard the message." But furthermore, it also provides a way to tell the responsible party that the message failed a check. For example, if security@paypal.com fails a DMARC check (either through SPF or DKIM), the email receiver can send the message to an email address that says "Hey, this message failed an SPF check. Was it legitimate or not?" If it is a false positive (perhaps a new server brought online), Paypal can add it to its SPF check. If it's a phishing message, Paypal can investigate to have the website taken down.

The strength of DMARC is that it is a stronger way to protect a brand from being abused; receivers can discard spoofed messages and senders can figure out just who, exactly, is sending mail as them.

The weak point of DMARC is, unfortunately, the weak point of SPF and DKIM — spammers and phishers don't need to spoof a domain in order to fool users into taking action. If a spammer sends mail from security@paypal.com.yakzas.com (a fictitious domain), many users just see that first part (paypal.com) without being more aware that there is more to the message.

And if a phisher signs up for a cloud service that issues temporary credentials, they can create the account paypale.onmicrosoft.com and send spam from there to avoid IP reputation blocking (and to the spammer that is abusing our Office 365 service, we know what you're doing, you jackass) while hijacking the reputation of another brand in the From address.

The strength of DMARC is not so much that it combats phishing but that if a good domain is authenticated, mail user agents (like Gmail, Hotmail, Outlook, etc) can highlight that the sender is a trusted sender and highlight it in blue or put a little icon beside it. Since users use visual clues to make heuristic decisions, the lack of a trusted symbol can train people to be suspicious.

Anyhow, it's nice to see that the authentication/validation protocols are consolidating.

Written by Terry Zink, Program Manager

The made up addresses are largely dictionary attacks, which is obvious when I see sequential spam to barry@, betsy@, and bruno@. Some of them are company addresses that leaked to spammers before the companies went out of business years ago. And some are just mysteries.

My friend Bob Frankston has had his own vanity domain since 1992, which gets a lot of spam to spamtrap addresses. I automatically diagnose and send off abuse reports for a lot of it. Today I got a hand written response to one of them from a database marketing company in Florida. It said, in part:

This email resolves to a master record for [a name and address of a guy in Pennsylvania].

The recorded was added to the client's file on 11/12/2002 per a trip preference card that was sent to the postal address listed above. The trip preference card asks where someone would like to travel, and for their email address to be sent notifications.

If [that address] had changed their mind about receiving emails, we diligently suppress/remove opt outs. However, I do not see that email in our suppression, opt out, or feedback loops.

That wasn't too surprising, I've gotten other mail to that spamtrap from other spammers who gave me the same guy in Pennsylvania, who has no relation to Bob, and it's barely possible that someone could have scribbled something on a postcard that might have been mistranscribed as the spamtrap address, although the name of the alleged subscriber has no visible connection to the spamtrap address either. It's certainly plausible that once someone had the bad info, they sold it to lots of other marketers.

But two things jumped out at me. The first is the date, 2002. They've been spamming this address for ten years. Since it is a spamtrap, it has never responded, never ordered anything, never "opened" a message (ESP-speak for fetching the URLs in the message.) But they keep pumping out the mail anyway. The competent ESPs I know all purge their lists of dead addresses eventually, certainly in a lot less than ten years.

The other is the inability to imagine that every address in their crummy database isn't a live potential customer. This address never "changed their mind" because it doesn't have a mind. It's a spamtrap. It sends no mail, and it won't opt out because it never opted in.

I wish this situation were atypical, but it's not. If the putatively legitimate e-mail marketing industry wanted to understand why they've earned such a poor reputation, it wouldn't be hard to figure out.

Fun fact: Bob's last name happens to be the name of a town in Australia. Someone there has misconfigured one of their systems to send status reports with personal information about their clients to yet another made up address in Bob's domain, which I expect is totally illegal under Australian privacy law. I haven't been able to stop that, either.

Written by John Levine, Author, Consultant & Speaker

This post is about IP address reputation with some background on why IPs are so important and why ISPs focus so heavily on the sending IP.

Why IP addresses?

ISPs built reputation around IP addresses because it was one bit of data that malicious senders / spammers couldn't forge. The connecting IP is a fundamental part of the network transaction and if you forge an IP then SMTP can't work. Because that was the reliable data they had to work with, that's what they used. Even now, when there are other kinds of data, the IP address is still the first thing the receiving MTA sees.

What is IP reputation?

IP reputation can best be summed up as "past performance is an indicator of future results." In other words if recipients responded well to mail from an IP address in the past, then they're likely to respond well to new mail from that IP address.

How is IP reputation measured?

While each spam filtering company and ISP have their own ways of calculating the reputation of an IP address, there are some similarities in what they measure.

• How many non-existent email addresses is this IP attempting to deliver to?
• How many abandoned email addresses is this IP attempting to deliver to?
• How many "known bad" email addresses (spamtraps) is this IP attempting to deliver to?
• How many recipients complain about receiving this mail?
• How many recipients complain about not receiving this mail?
• How respectful of my resources is this IP?
• Does this IP keep connections open for long periods of time?
• Does this IP retry deliveries too aggressively?
• Does this IP stop mailing addresses after receiving a "user unknown" message?
• Is this IP address configured as if the associated machine was infected by a virus?
• Is this IP address listed on blocklists we use?
• That is by no means an exhaustive list of what ISPs measure. If they can measure it they've tried. If the measurement helps them separate spam mail from not-spam mail then they're using it.

How fast does IP reputation change?

IP reputation is often measured over multiple time periods. ISPs can look at a 1 day, 7 day, 30 day and 90 day reputation. A good analogy is stock prices. Prices can be very volatile in the short term, but more consistent over the long term. A single bad day, where one or more reputation measurements go bad, may affect delivery that day or the next day but won't damage an overall good reputation. Likewise, a few days of improved mail may not be sufficient to counter months of poor reputation.

How is IP reputation used?

Mail from IPs with a high reputation is accepted faster and at a higher rate than mail from IPs with a lower or unknown reputation. IP reputation can also influence whether mail is delivered to the inbox or the bulk folder.

Key IP Reputation takeaways

• IP reputation is about how recipients react to mail from that IP. Happy, content recipients turn into good delivery.
• Brief changes (for good or bad) don't necessarily ruin delivery over the long term.
• Steady improvements will result in improved reputation.
• It may takes as much time to change a reputation in one direction or another as it took to establish the reputation in the first place.

Written by Laura Atkins, Founding partner of anti-spam consultancy & software firm Word to the Wise

ACMA receives a major compliment

In 2004, when I first entered the anti-spam arena, this was a mantra that I had to hear very often: "Spam is international. We cannot do anything", spoken with a lot of emphasis and some despair. Unfortunately in 2012 this is still true for many countries. Not because of the fact that it is impossible to do something about spam, no, but due to a lack of initiatives. I think that a great compliment to Australia's ACMA (Australian Communications and Media Authority) was published on CircleID in a comment to an article about the impact of Canada's spam law on local businesses. Brett Watson, an Australian internet engineer, writes:

"However, my present (and general) lack of anything to complain about reflects well on the law and its enforcement… Perhaps what's most telling is that I have, for the first time, subscribed to some advertising newsletters in recent years. I don't feel the need to jealously protect my email address any more, or diligently use uniquely tagged addresses when handing them over. I trust ACMA to keep the companies in line, and the trust seems well placed so far."

This proves that fighting spam is effective and that the combination enforcement with filtering by ISPs keeps mailboxes clean. Spam hasn't gone away, but at national level companies are disciplined and mostly act within the law in the few countries with vigorous enforcement bodies.

Who enforces what?

Privacy and spam are closely related. Spam is seen as an invasion of privacy. But it goes way beyond mere privacy. Privacy sensitive data is often used, sold or worse stolen in order to approach people. Whether to sell a(n illegal) product, phish for more (bank)data or industrial espionage, a stolen e-mail address is often the basis of law violations. The patchwork of enforcement agencies, unclear enforcement powers, the lack of understanding of the issues at stake, of resources, training or powers, the unavailability of online reporting of spam or cyber crime, all make that enforcement is far from optimal in most countries.

Standardisation of spam and cyber crime law

Could a standardised law, with a standardised toolkit for enforcement agencies make a difference? Yes, I think that it would. For the public it would mean that there is the certainty that when the law is broken, it is clear who to report to and that it is likely that an investigation follows. That it makes a difference to complain. For senders it also sets clear boundaries. Their business continues, as is proven in e.g. The Netherlands, but in compliance with the law. Next to that it offers this clearness in 27 states.

As spam, e-fraud, phishing, cyber crime and worse are all so closely related and often involves several countries, it makes sense to be more directive from Brussels. At national level there are so many different laws, ministries and enforcement agencies involved, that coordination there is almost utopian. Next to the fact that success without industry participation is clearly unthinkable. Despite the fact that the Dutch National Cyber Security Centre is a promising initiative, it is obvious that for most countries this form of public-private cooperation is hard to attain.

A proposed course of action for the EU Cyber Security Centre

The discussion about the EU Cyber Security Centre is under way. Let me give a pointer on what the centre could do. To my mind it ought, also, to actively collect, analyse and share data with those involved: public and private entities, universities. This gives the centre coordinative powers in matters cross border and across different enforcement organisations as well. Two difficult hurdles taken… should this come to pass. The combination of the overview and oversight with the transparency caused by available, shared data makes all concerned answerable for their (lack of) actions to the centre and each other. I am also convinced that this model will lay the foundation for cooperation with whole new groups of Internet industry partners that are now harder to reach/convince.

Ambition at Commissioner level

If Commissioners Kroes, Malmström and Reding used their powers to harmonise the laws and enforcement in the way Ms. Reding proposes for privacy, i.e. the same law and enforcement tools, standardised enforcement agencies and a point of case handling, the fighting of privacy infringements, spam, malware and cyber crime may actually take a turn for the better. They are so intertwined that another approach is (well, should be) almost unthinkable.

The combination of a pro-active EU Cyber Security Centre with a layer of harmonisation where enforcement is concerned will prove to be a structural step forward from the present situation in many countries. Yes, this is ambitious, but it is clear that the present approach is not going to change much. Everything cyber is still a field day for criminals and a private company, Microsoft, so far is the most successful in fighting botnets. This ought to be different, shouldn't it?

Written by Wout de Natris, Consultant international cooperation cyber crime + trainer spam enforcement

Known as CASL, the new law aims to crack down on spammers and mailing list companies but in doing so, tightly regulates the way businesses can market to prospective customers via email and online.

In a nutshell, CASL requires a business to obtain consent from the recipient before it sends out commercial electronic messages (CEMs). It isn't limited to email; consent must be given for any electronic message, which could also include messages sent via social media, text messaging, instant messaging, sound or video. If your business operates outside of Canada, you shouldn't assume the Anti-Spam Act doesn't apply to you. If a computer system within Canada is used to send, receive or even route the message, then the law could also apply to you.

It is in obtaining consent before sending an electronic message where the Canadian Anti-Spam Act differs from its American equivalent. The United States' CAN-SPAM Act requires that recipients are given an opt-out option from commercial messages but under CASL, recipients must opt-in to receive electronic messages.

The fines for violating the Anti-Spam Act are hefty. The maximum penalty per violation for an individual is CAD $1,000,000 and $10,000,000 for corporations. With potentially crippling fines waiting in the wings for violators, how can you ensure your company is compliant?

The first thing is to be aware of which messages require consent before they are sent. There are a few exceptions, which include personal relationships or when the company is providing requested information. Consent can usually be implied if there is an existing business arrangement of two years or more, or if an email address has been disclosed in the course of business. You can read more about exceptions to CASL here.

If your electronic message doesn't fall under an exception category, then you will need to obtain consent before sending it. The message should also include an unsubscribe mechanism. To ensure compliance, your company should establish procedures to obtain consent for electronic messages and educate staff on the Anti-Spam Act. The most important thing to remember before you press 'send' is the onus is on your company to prove you received consent.

Do you operate a business in Canada? How do you think the Anti-Spam Act will affect the way you market electronically? Please contribute to the conversation below.

Sources:
Canada's Anti-Spam Legislation: Casting a Wide Net
Anti-spam law draws backlash
Three 2011 developments that changed your inbox forever
Canada: Preparing For Canada's New Anti-Spam And Online Fraud Act
CAN-SPAM Act: A Compliance Guide for Business

Written by Susanna Sharpe, Social Media Manager

1. You will make your own energy: Anything that moves has the potential to create energy. Your running shoes, your bicycle and even the water flowing through your pipes can create energy.

2. You will not need a password: Your biological makeup is the key to your individual identity, and soon, it will become the key to safeguarding it.

3. Mind reading is no longer science fiction: Scientists are researching how to link your brain to your devices, such as a computer or a smartphone, so you just need to think about calling someone and it happens

4. The digital divide will cease to exist: In five years, the gap between information haves and have-nots will narrow considerably due to advances in mobile technology.

5. Junk mail will become priority mail: Think about how often we're flooded with advertisements we consider to be irrelevant or unwanted — it doesn't have to be that way anymore.

I want to start with the last one — that junk mail (i.e., spam) will disappear. You'll need to watch the video to get the nuances of the prediction, but IBM says that in five years, Junk Mail will become a thing of the past. Instead, what will happen is that spam filters will become so good at knowing what type of mail you want to receive that it will filter out everything that it knows you don't want and deliver you the mail it knows that you will want. In this way, junk mail becomes priority mail.

Imagine that your phone syncs up somewhere and sees that your favorite band is coming to town. Your personalized spam filter would know that you like this band and either (a) allow emails like this to pass through your spam filter to your inbox or (b) actively go out and find the information, delivering it to you.

How likely is this to occur?

Bold predictions about spam filters have occurred before. As everyone likes to point out, Bill Gates predicted in 2004 that spam would become a thing of the past. Yet here we are, 8 years later, and spam is still a problem.

But it's not the same problem that it was before, now is it?

Let's take a look at this. The spam problem — in email — has changed over the past few years. We used to see a lot of botnet spam with illegal content but we see much less botnet spam these days. If you read any report about the state of spam, you'll know that it has declined considerably over the past year. However, what has replaced it (in terms of how annoying it is and how many complaints it generates) is snowshoe spam which is smaller and lighter and looks a lot like marketing mail. My prediction is that the next big revolution in antispam technology is figuring out a way to effectively deal with snowshoers (now that we've gotten pretty good at stopping botnet spam).

Snowshoe spam is annoying. But, if spam filters do get good at stopping snowshoe spam, in addition to remaining good at stopping botnet spam (or botnet spam stays down), then IBM's prediction becomes possible. Just think about it for a moment:

• Spam filters are good at blocking most spam so few people get it.
• Spam filters are good at detecting legitimate marketing mail.
• Social networks and search engines are becoming more and more personalized. When you login to Facebook, the ads are targeted to you. If a spam filter talked to a social network, then it would be able to automatically decide which marketing mail to get to your inbox based upon a best guess of the things you are interested in. For example, my wife and I regularly attend lecture series put on by National Geographic. If I "liked" National Geographic on Facebook, then if they ever sent mail to me, my spam filter (after talking to Facebook) would let the mail through to me. And I'd say "Hey, this upcoming talk looks pretty interesting!"
• The principles we have learned over the past 10 years still apply. A spam filter would guess what the person would like to see, but the senders of the mail still need reputation to ensure their delivery. They'd need to sign their mail with DKIM, publish SPF records and have low levels of spam complaints, ensure opt-in best practices, and so forth.

Thus, the next big trend in spam filtering, according to IBM, is theoretically possible. Is it possible to do within 5 years?

Maybe.

Spam still hasn't been totally solved. Just in the past week we've seen an eTrade spam blitz and then a Bank of America spam blitz, and these were cases of botnet spam with relays behind relays. We haven't managed to eradicate that type of spam yet, but it isn't the problem it once was.

But looking over how something like this might be accomplished, it's not tough to visualize. Imagine someone (let's say me) had a Gmail account and used their Google+ account actively. If they +1'ed things like Minyanville and TheStreet.com, and then went into Gmail and said "Bring me stuff that's relevant," it's not difficult for Gmail to sift through their mountains of mail and bring you relevant things.

But on the other hand, there's the problem of permission. Would you want Gmail to give you updates from financial services (for example) that you never subscribed to? For me personally, if I owned stock in Apple, I might want news alerts brought to me even if I never wanted to hear everything from The Motley Fool. But perhaps I'd want to hear everything from Minyanville when they talk about Apple. After all, I +1'ed Minyanville so I must like it. But Minyanville only sends mail to people who signed up. But I want Gmail to bring me stuff that is relevant. What do I do?

I'm sure people will figure it out eventually. It's an area that is ripe for exploration.

Written by Terry Zink, Program Manager

Canada's antispam law is much tougher than most jurisdictions. Aside from the penalties of the law, which are steep, what differentiates it the most from the US law is that Canada's law is an opt-in law; marketers who send commercial email must be able to demonstrate that they received consent in writing in order to market to people. It also allows for the international sharing of information and evidence to pursue spammers outside of Canada. You can view the summary here: Industry Canada summary of bill C-28.

As expected, people who are most affected by the law — marketers — are upset about the lack of wriggle room and how it could affect their business. From the Canadian lawyer mag:

"This legislation has quite a broad scope and it will capture many regular business operations by legitimate operators, so in-house counsel are starting to realize that this imposes additional requirements that they're going to need to take into account."

"The problem is since the regulations haven't been finalized, we can't put in place the final processes, training, and implementation plans."

"Unfortunately, when the draft regulations came out, frankly they didn't do anything about the concerns that industry mentioned about the scope of the legislation itself."

"Companies are put in a position where they have to go out and re-qualify their entire database."

In other words, companies thought that the antispam law went too far because they are used to sending out email without getting explicit consent from the people they email.

The fact of the matter is that it's not all that difficult to build a list by consent. Double opt-in (where you get people to click a box saying that they want to opt-in, and then send them email to click the link to confirm) is the easiest way to do it. Single opt-in is the second easiest.

The reason that some companies would complain about the broadness of scope of the law — which is intended to target spammers — is because a lot of companies buy email marketing lists from other parties. When someone says "We may use your information to sell to other partners," this is a quick-and-easy way of building a mailing list. The CASL negates this practice, which was never ethical to begin with.

Yet the most interesting quote of all is the following:

Michael Osborne, a partner at Affleck Greene McMurtry LLP, says the tumult around CASL is unlikely to end with its coming into force. He calls the legislation Canada's "biggest restraint ever on freedom of speech" and says he expects various provisions to be challenged in court. "If I want to send a flyer to your house, I can do it. That's part of the trade-off for living in a free economy, and I don't see why e-mail would be any different. I don't like getting junk at my door or in my inbox, but if I want to live in a free economy, I think I have to accept that," Osborne says.

The comments above are puzzling because I don't think anyone actually thinks legislation against spam infringes on freedom of speech, they have a weird view of it. Freedom of speech guarantees that you can communicate your message but it does not mean that others have to listen to it. People in the US have challenged antispam laws, claiming it violates their freedom of speech but courts have never supported this argument.

But the real puzzling part is the bolded text above. It's true, if you want to send me a flyer, you can. But this is not the same as email because if you want to send flyers, you have to pay the postage, and you have to pay for creation of the flyer. You only have limited resources and so you are careful about it. With email, the cost of it is borne by the recipient. It is almost as easy as it is for you to send 100,000 messages as it is to send 100. It is your recipient who has to deal with the cost of network storage of bandwidth.

We view our email inboxes as an extension of our homes. You can deliver a message to my mailbox but you cannot come into my home and force me to listen to it. I have a right to privacy and unsolicited email violates that. I guess this is getting into Constitutional arguments now (and I am poorly equipped to expound upon them).

The fact is that flyers and email are not the same. They are quite different. That's why computers are so great, you can do so much more with much less input.

Sometimes lawyers say the weirdest things.

Written by Terry Zink, Program Manager

Abuse-POC, a.k.a. abuse-c or abuse-mailbox entries are the subject of ongoing developments at AfriNIC, APNIC, ARIN, LACNIC, and RIPE. It may take a while for Regional Registries to converge and complete their work. Abusix.org offers an Abuse Contact DB that can be queried via DNS, until then.

Some network providers allow clients to specify abuse-mailboxes along with other contact info, while others don't. They don't seem to be striving to act as the Internet police. They operate according to their commercial policies, albeit they try and comply with local laws. Thus, mailbox providers don't always have full control on what gets published on the number databases, independently of their behavior.

On the other hand, the DNS was conceived to avoid reliance on numbers and use names instead. The advent of IPv6 may exacerbate that principle. Techniques like DKIM and SPF allow to associate a domain name to a mail message. Such techniques are mature enough to yield results that are more reliable than those supplied by rDNS, which suffers the same limitations of control as number databases. However, there is no standard way to learn whether a domain offers a feedback loop, or what is the email address to be used for (automated) abuse reports. The only hint is abuse@domain, as specified by RFC 2142, which can be deemed heuristic at best. (Contrast that with the fact that abuse-c seems to be going to be mandatory, and that providing false contact data may lead to deregistration of IP blocks, at least at some RIRs.) Abuse.net offers a name-to-abuse-mailbox functionality, but there is no prospect similar to the number case, yet.

There is an IETF working group,

Messaging Abuse Reporting Format (marf)

the same that standardized the ARF format, that might make some decisions about standardizing such reporting-discovery functionality. However, the working group experienced a drop of participants recently, for various causes. The likelihood that it will complete its work is getting lower and lower, unless new people will want to review its drafts and post comments on its mailing list. If this is a call for participation, you have been called!

Written by Alessandro Vesely, Tiny ISP and freelance programmer

They observe that bots typically live on cable or DSL connections with slow congested upstreams. TCP sessions from bots turn out to be fairly easy to recognize by RTT, window, and retransmits, something that people have known at least since a paper at the 2008 CEAS conference on the topic.

This paper tries to see whether it would be practical to use that info to manage spam in real time. They have a network analyzer called SpamFlow that figures out per-connection characteristics. Then as a proof of concept they wrote a Spamassassin plugin to train on the data from SpamFlow and try and do filtering. They do some sort of hand-wavey load testing to see whether SpamFlow can keep up with a realistic mail load, and if it trains fast enough that it would provide useful data in real time. They claim that their results show that it does both.

It's not obvious how best you would use this in combination with all of the other anti-spam tools people we have, most notably blacklists like the CBL that very accurately identify IPs of botted hosts by looking at the characteristics of mail received at large spamtraps. One thing that occurs to me is this sort of thing might be useful if mail moves to IPv6, since building v6 blacklists will be hard due to the size of the address space, while this lets you estimate the bottiness of each connection directly. Also, rather than accepting or rejecting mail, you might slow down mail reception from hosts that seem to be bots, both to give preference to non-bot senders, and because bots tend to be impatient so if you slow down a dubious connection and it gives up, it was probably a bot. The Turntide appliance did something similar five years ago, although it used different heuristics for deciding what to slow down.

This technique looks only at the characteristics of the TCP session, and not at the contents of the session, which means it also doesn't look at the contents of the messages. It might be useful in contexts where for legal or political reasons the spam filter isn't allowed to look at the messages, but users want spam filtering anyway. The authors point out that it is in principle applicable to any TCP transaction, so it might be useful against web queries from bots, too.

It's hardly a FUSSP, but it's an interesting paper.

Written by John Levine, Author, Consultant & Speaker

Another theory about greylisting is that if you defer mail from a new IP, by the time the sender retries, if it's sending spam it'll have hit spamtraps and been added to blacklists. I recently realized that I have enough log data to check that theory, so I collected some statistics for the past week, which is as long as I keep logs about mail connections from blacklisted hosts. The IPs I greylisted broke down like this:

	Count	Percent
No retry	3,803	35.8%
Retry too soon	3,345	31.5%
One retry	1,183	11.1%
More than one message	1,635	15.4%
Blacklisted	561	5.3%
Retried, blacklisted later	89	0.8%
Total	10,616	100.0%

No retry and Retry too soon are senders that greylisting kept from sending anything, again, about 2/3 of mail. (My greylister requires that the sender wait at least a minute, since some spamware sends several messages a few seconds apart.)

The next two are senders that retried successfully and sent one message, or more than one message. (If a sender retries too soon, then retries again after more than a minute, it's counted in one of those two categories.) Blacklisted means that when the IP retried, the IP was on one of the a blacklists I use, in nearly all cases Spamhaus Zen. The last line is IPs that retried successfully, but were blacklisted when they tried to send other messages later.

The 5.3% for Blacklisted probably overstates how much mail was caught by waiting to see if an IP was blacklisted. My logs don't say whether the delivery attempt that was blacklisted was trying to deliver a message with the same To and From addresses, in which case it would have been delivered, or a different message, in which case it would just have been greylisted again. Spot checking shows IPs that were greylisted repeatedly, before appearing in a blacklist, which suggests that they were sending different messages.

Also, for the few IPs that were blacklisted later, they were generally blacklisted much later, hours or days later, far longer than any reasonable greylisting strategy would force mail to wait.

So greylisting still works, but it's almost entirely because spamware doesn't retry, not because it gets blacklisted.

Written by John Levine, Author, Consultant & Speaker

We've now been using greylisting for close to a decade, and some people have argued that it's no longer useful, since the bad guys could easily fix their spamware to retry, or since bots are so cheap, they could just send everything twice. So does it still work?

I recently went through my greylister's logs and collected some statistics for both a recent week, and the past year, about hosts that I greylisted:

	Week	Year
No retry	12121	294812
One retry	7456	62402
Many messages	4956	74590

The first row is the number of hosts that got a soft fail and never came back. The second row is the number that retried the message that failed, but never sent anything again, and the third row is the number that retried and sent more messages after that.

As you can see, for the week, about half of the greylisted hosts didn't retry, and over a year, about 2/3 didn't. That's still a lot of mail my mail server didn't have to filter. I attribute the different ratios to the shutdown of several botnets over the past year, evidently botnets that didn't retry.

So it's certainly not a magic bullet (what is?) but greylisting still is an effective way to deter a lot of spam cheaply.

Next, Greylisting Still Works - Part II

Written by John Levine, Author, Consultant & Speaker

The new MainSleaze blog at http://mainsleaze.spambouncer.org/, run by long time anti-spam activist Catherine Jefferson is all mainsleaze all the time, and she's having no trouble finding plenty of examples.

The problem with mainsleaze is that it is generally mixed in with mail that the recipients asked for, and there's no way to tell the difference mechanically. Since it is legal in the US to send spam until people tell you to stop, although it's against the terms of service of every ISP in the country, poorly informed or ethically challenged marketers beef up their lists by buying lists or by e-pending, trying to guess the e-mail address of customers for whom they have other contact info. Or sometimes, they decide to reactivate lists of addresses so old that some of them have been abandoned and later reassigned to other people.

As a result, if a mail system filters out all the mail from a mainsleazer, they'll get complaints from the people who signed up. If they don't filter, they'll get complaints from the people who didn't. Most mainsleaze is CAN SPAM compliant, so if you tell them to stop they generally will, at least until they buy another list with your address or e-pend it from someone else.

One ray of hope is the new Canadian anti-spam law, now expected to come into force in early 2012. It requires that commercial e-mail be sent only to recipients who have asked for it, or who have a demonstrable existing relationship, that is, no mainsleaze. Any large mailing list in the US is almost certain to contain addresses that are delivered to mail servers in Canada, either of Canadians (many of whom do not have .CA addresses), or of Americans who use a mail service hosted in Canada such as Tucows' hosted e-mail.

If a mailbox is in Canada, Canadian law applies, and the new Canadian law allows spam recipients to sue the sender, even if the sender isn't in Canada. So mainsleazers who don't clean up their acts are likely to be on the receiving end of some expensive lawsuits. With any luck, after a few settlements, they'll start to get the hint.

Written by John Levine, Author, Consultant & Speaker

"Email appending" is also known as "e-appending" or "e-pending." As used in this document, it refers to taking known demographic information and using various methods to determine an email address for the purpose of adding people to a list or otherwise sending them email messages.

This definition is alright but I didn't find it as helpful as it could be. I looked it up on some other sites and I have a better description.

Suppose you are a marketer with a list of people and their mailing addresses. These are people who like to receive information about upcoming flights on airlines because they travel a lot. Suppose your list looks like the following:

Fred Flintstone Fabrikam Industries 175 NE 22nd Pl Flint, Michigan	Frank Grimes Tesla Productions 725 Evergreen Terrace Springfield, Illinois
John Bauer Woodgrove Bank 4888 Cowell Bay #12 Key West, Florida	Jeff Johnson Nert 61221 W Jackson St Arcadia, California

Every once in a while, you send them deals from Alaska Airlines about a great business package from Atlanta to Seattle. Or you send them a deal from American Express offering them two round trip tickets anywhere in the US for the price of one (but the one ticket costs as much as two).

The problem is that sending stuff over mail is slow and the response rate isn't great, and as Netflix has so eloquently reminded us, the physical media business is dying. Also borrowing from the Netflix model, you decide to annoy your customer base by changing something that wasn't broken and attempt to get into the email communication business.

You go out and find an online marketer who has a list of email addresses. You take a look at their list and it contains the following names:

ksmith@fabrikam.com
john.baker@woodgrove.com
maddox@weevils.org
jj@nert.com
fgrimes@tesla.net
jimmybindlesnoot@tesla.net

You browse through the list and while some of the email addresses on there might be associated with someone on your list, you don't see anything stand out… until you get to fgrimes @ tesla.net. Wait a minute… you just so happen to have a Frank Grimes who works at Tesla in your database. Could they be one in the same? Well, there's a pretty good chance of that so you take fgrimes's email address and update your database, putting it into Frank Grimes' contact information.

Having gotten into the email business, you send out a communication to fgrimes@tesla.net containing the next airlines deal. What a cheap and inexpensive way to build a mailing list!

Except that fgrimes is not Frank Grimes. The email addresses belongs to Foster Grimes, the director of IT in Tesla. Foster doesn't take too kindly to receiving mail to his email address when he never opted into it… he knows he has never opted into it and takes steps to block the unsolicited commercial email. Your marketing company can no longer deliver to Tesla. You're lucky that this email address wasn't a honeypot.

In this sense, building an email list is prone to errors. It doesn't follow that people with similar names are the same people. There is a probability of it, and it may be greater than 50% odds, but it is closer to 50% than it is 100%. If you are going to scrape lists like this, you will end up with wrong email addresses for a large proportion of your database, and you will be sending unsolicited commercial email.

Furthermore, even if you do get it right, it doesn't follow that someone who has opted in to regular mail has also opted in to email. I get all sorts of stuff in the mail I don't want. I get offers from credit card companies all the time, and grocery stores all the time, and religious organizations all the time. I don't remember opting in to any of them, but for the sake of discussion, let's say I did. Thank goodness I didn't give them my email address! I do not want them sending me email and regular mail! No way!

If I give you my mailing address, then you have permission to send me mail. That's it. You do not have permission to contact me by any means you wish, you have permission to contact me by the method I say you can. That's all. No more (and I wouldn't mind less, either). If you go out of your way to hunt down my email address, good for you. But you can't use it to send me email because I never said you could send me email. I know how I want to be contacted, and I told you.

And that's it. I want to control what mail comes into my email inbox. If I didn't say you could and you send me advertising mail then I would probably consider you either a spammer or a bulk mailer wearing a dark gray hat.

That's why email appending is bad.

Written by Terry Zink, Program Manager

Last week, a bunch of people noticed that Yahoo was blocking mail containing references to a protest against Wall Street. This understandably upset people who were trying to use email as a communication medium. Many people decided it was Yahoo (a tool of the elites!) attempting to censor their speech and stop them from organizing a protest.

Yeah. Not so much.

Yahoo looked into it and reported that the mail had gotten caught in their spam filters. Yahoo adjusted their filters to let the mail through and all was (mostly) good.

I don't think this is actually a sign of filters being broken. The blocked mail all contained a URL pointing to a occupywallst.com. I know there was a lot of speculation about what was being blocked, but sources tell me it was the actual domain. Not the phrase, not the text, the domain.

The domain was in a lot of mostly identical mail coming out of individual email accounts. This is a current hallmark of hijacked accounts. Spammers compromise thousands of email accounts, and send a few emails out of each of them. Each email is mostly identical and points to the same URL. Just like the protest mail.

There was also a lot of bulk mail being sent with that URL in it. I've been talking to friends who have access to traps, and they were seeing a lot of mail mentioning occupywallst.com in their traps. This isn't surprising, political groups have some horrible hygiene. They are sloppy with acquisition, they trade names and addresses like kids trade cold germs, they never expire anything out. It's just not how politics is played. And it's not one party or another, it's all of them. I've consulted with major names across the political spectrum, and none actually implement best practices.

As I have often said the secret to delivery is to not have your mail look like spam. In this case, the mail looked like spam. In fact, it looked like spam that was coming from hijacked accounts as well as spam sent by large bulk mailers. I suspect there was also a high complaint rate as people sent it to friends and family who really didn't want to hear about the protests.

To Yahoo!'s credit, though, someone on staff was on top of things. They looked into the issue and the filter was lifted within a couple hours of the first blog post. A human intervened, overruled the algorithm and let the mail out.

I bet this is one of the few times anyone has seen that Yahoo does outbound filtering. Given it's a politically charged situation, I can see why they assume that Yahoo is filtering because of politics and censorship. They weren't though.

Written by Laura Atkins, Founding partner of anti-spam consultancy & software firm Word to the Wise

First contact

Over four years ago the first contact between OPTA (the Dutch telecoms and post regulator) and the RIPE NCC was laid. It was an awkward meeting of two groups of people who were talking to each other, but didn't connect at any level. Around the same time RIPE NCC got into contact with LEAs that had an urgent need for accurate information in cybercrime investigations and was confronted with a growing number of requests for information. All this culminated in invitations participate in the Government Roundtable meetings for LEAs, including a special meeting aside.

From misconceptions to dialogue

At the first meeting in 2008 it was clear that there were several misconceptions on both sides as to content and purpose. There was a distinct tension between both parties and an expressed urgency on the side of the LEAs, which made dialogue hard to establish. What happened however, was that the friction built up during the first meeting was taken away by discussing possible future approaches. An agenda of topics was identified and an invitation formulated to continue the discussion. This led over the course of 2009 to the invitation to participate in respective relevant events. OPTA and representatives of cybercrime units presented at RIPE meetings, while RIPE NCC presented at the London Action Plan, the e-crime event in London and in the EU Cyber Crime Task Force. Relevant knowledge and information was shared between both sides.

Understanding each other's positions

This made a few things clear to participants. Law Enforcement officers learned to understand about policy processes within the RIPE community and that the only way to influence these processes is to participate and address the right people. Also they learned what sort of organisation RIPE NCC is and the sort of information she has on its members and the Internet in general. What of this information is public and what is private sensitive data. RIPE and RIPE NCC learned that (governments and) LEAs have legitimate concerns about the safety and security of the Internet and need accurate information on LIRs for investigating criminality or spam violations. But most probably also that they have no use for members that do not pay their bills and are untraceable as well as that it is not good for reputation when RIPE NCC is, no matter how unwillingly, associated with (organised) crime, which unfortunately, in very small numbers, was the case. This awareness caused RIPE NCC to look at her standards and procedures and alter them when deemed necessary, which is a great claim for self-governance.

Cooperation is successful

The reaching out led to the installation of the Cyber Crime Working Party in May 2010. In the CCWP LEAs, the Anti-Abuse Working Group of RIPE and LEAs share data and work together. The biggest challenge for law enforcement agencies is to dedicate resources to cooperation and participation, that do not immediately show a result in facts and figures presentable to the outside world. The figures presented by RIPE NCC on 12 September, as well as recent presentations by ARIN and AfriNIC, show that cooperation does pay off, but needs time to develop. On both sides! So use the figures within your respective organisations and make sure that this raises the awareness of the right people there.

As CCWP chair I complemented RIPE NCC with these results and noted that we have come a long way over the past few years. There is more to be done, but this moment of reflection is as good as any.

Written by Wout de Natris, Consultant international cooperation cyber crime + trainer spam enforcement