|
||
|
||
Below is a chart that shows the amount of inbound mail that we see, both spam and non-spam, over the past three and a half years. This data also appears in the Microsoft Security Intelligence Report, but the data there is monthly (or half-yearly) whereas this data is weekly:
Microsoft Forefront Online – Total Weekly Spam (red) and Non-Spam (blue)The charts are normalized to show the scale (i.e., the left hand scale is not 35,000 messages, but is 35,000 x some number). In addition, the spam in red is plotted against the primary (left) Y-axis and the good mail in blue is plotted against the secondary (right) Y-axis.
You can see in the above that the amount of good mail that we see has continued to increase over time. This is because of an increased customer base, not because the total amount of good mail worldwide has gone up (although it has increased marginally as more and more people start using the Internet). However, the amount of spam has plummeted from 23,000 in mid 2010 to 5000 now, a drop of over 75%. The contrast couldn’t be starker—spammers are not spamming as much anymore.
It almost looks like the battle against spam is almost over. What’s still left to do?
Here’s a couple of things that are unique to spam and not other forms of communication:
Eventually the first two will be handled. Pesky bulk mailers will see their reputations dwindle down to nothing and they will get added to blocklists along with everyone else. The second will be handled in the same way—as the spam traps start to attract more and more foreign language spam, they will populate their lists from URLs pointing to Portuguese spam sites, or IPs sending high volumes of spam.
The third is the most difficult. Filters will continue to update quickly but products other than spam filters will be required in order to prevent these, such as traffic analysis tools and intrusion detection software. That will open up a whole new niche for security vendors but will likely be plagued by even less collaboration than there is now (would Microsoft want to share their infrastructure layout with Google? I think not, nor vice versa).
That will take some creative thinking and is probably the next big trend in security.
Sponsored byVerisign
Sponsored byIPv4.Global
Sponsored byRadix
Sponsored byDNIB.com
Sponsored byWhoisXML API
Sponsored byCSC
Sponsored byVerisign
A World-Renowned Source for Internet Developments. Serving Since 2002.
I have some reservations about your assertions and conclusions
First off
Well, the decapitation of several well-known botnets like Coreflood, Mariposa Botnet, Rustock and Bredolab may well mean spammers aren’t spamming as much in the past because they can’t, for the moment. But, we have also seen several huge and sustained spikes on other bots as recently as last week.
Almost, but not quite. I’d be interested in seeing figures from your systems regarding stuff blocked at the network boundaries by other means - IOW stuff that never counted as email, but merely a rejected connect.
Terry, you are speaking from the perspective of a corporate spam filterer (and yours is a perspective limited to ‘what you see’ which may or may not be what others similarly charged are dealing with), which is fine, but what do the ISPs report? How about the huge Freemailers like Yahoo!? We also know that many spammers don’t bother with corporate systems, they don’t even bother with anything else but Yahoo!/Hotmail/Gmail/AOL - nothing ‘in the wild’ to use their term.
Mostly, spammers want to get to Joe Enduser. To do that - they are now spamming Twitter & Facebook, and to a lesser degree LinkIn - does that not count as spam? Yes, closed systems, but just as much - a way to reach their intended target, illicitly.
I’d say the fight is far from over.What you are seeing is a lull in the action, and you may be away from the front. For the moment. It is normally said ironically, but in this case I do mean it guilelessly ‘Good luck with that’.
—
Neil Schwartzman
Executive Director
CAUCE : The Coalition Against Unsolicited Commercial Email
http://cauce.org
http://twitter.com/cauce
IM: caucecanada
TWO articles cross my desktop in less than an hour after seeing this posting
Massive Rise in Malicious Spam - M86
Exponential increase in malware-bearing spam spells trouble for IT
Statistically precise malware campaign promises headaches to the Nth power for IT this month
Neil, in response to your comments:
Coreflood was disabled in April, 2011. Rustock was disrupted in March, 2011. Bredolab was disabled in Nov 2010, and Mariposa was disabled in March 2010. The decline in spam began around April 2010, so some of these disruptions are after the major decline.
While it is true that taking botnets offline disrupts spam, it is more often than not a temporary reduction. The most obvious example is the McColo shutdown in 2008. Spam plummeted overnight but gradually built back up. Cutwail was disabled last year with the 3FN takedown but bounced back.
So while I would agree with your point that disrupting botnets can be a temporary takedown, the greater trend is a decline in spam from its peaks rather than a decline-and-subsequent-return-to-normal.
My chart includes rejected connections.
I agree it’s possible that corporate mail receivers are seeing different trends than free receivers. I would have to look up Hotmail’s stats, but all of the big receivers of mail who issue threat intelligence reports (Cisco, Symantec) are all reporting declines in email spam so my comments are inline with what the corporate email space is seeing.
I was excluding that type of spam from my post as I was talking about email spam specifically. This isn’t something I have written about recently but my position is that while email spam has declined, Internet abuse has not. It has merely shifted from one corner of the Internet to the other. Twitter, Facebook and LinkedIn are the “new” email and that’s where the spam has moved.
We do have a footprint that has both corporate and consumer email users, so I guess we have the best (or is it the worst) of both worlds in terms of the spam that we get. So, my two cents. I'd take a slightly longer term event horizon. That'd make your graph flatten out a lot, rather than the 2008-11 period which did go through a bit of a trough because of bot takedowns and the shift of at least some bots to attacking other messaging systems (social networks, IM etc). I am not sure if you counted all the spam that reaches your edge (and maybe gets dropped at a firewall etc) - that might skew the numbers a bit as well. And whatever spam does get through is rather harder to filter now than it was before, with the other side being just as busy reverse engineering filters and cycling through IPs as we are, blocking and filtering them. Generic bulk mail is getting tougher to filter because of snowshoe spammers evolving a lot. They evolve their rDNS patterns, their reputation monitoring and gaming, their content, their sending techniques etc, they set up numerous fake entities to acquire IP space from ISPs (even from RIRs, lots of them become RIPE LIRs under shell identities). So they need to be actively tracked and broad based blocks applied [maybe manually] to identify and cover their IP space when you start detecting lots of single IPs / clusters of IPs getting poor reputation. And your reputation systems themselves need to be monitored for signs of gaming and modified to penalize gaming. And there needs to be community engagement with ISPs, and with RIRs, to get things cleaned up from the supply side. As for the foreign language spam, we see a lot more of it because it is mixed in with legitimate traffic, sent through regular smarthosts operated by large ISPs in places like Thailand and Pakistan. Filtering the smarthost IPs would be a bit painful because your users and the provider's users who legitimately want to reach your users, will complain, and because the providers won't bother to react because of poor customer service. I'm not even counting the # of botted PCs listed in the XBL (and where countries like India, Vietnam etc are consistently in the top 5 because of a huge installed base, lots of users using outdated AV and bootleg windows etc), because we filter out XBL'd IPs.
Hi Terry,
Perhaps the most important thing to keep in mind is that while the absolute volumes of spam are dropping, it’s mainly because just one particular type of email spam, botnet spam, is declining.
A few years ago, botnet spam was around 95% of all spam. Infect a few thousand (or 10s of thousands) of machines, and spew away - there were few limits to scale, and the spammer doesn’t have to work very hard. Over the past few years, the effectiveness of botnet spam has declined because the ability of ISPs to block it has greatly improved to the point that more than 98-99% is blocked, plus the various take downs, means that the return on the investment of botnet development and operation has declined.
Now? It’s botnet percentage of the total is probably 80%. A decline from 95% to 80% may not seem like much, but in terms of absolute volumes, once you do the math, the total spam count goes down by a factor of four or more.
Does this mean that spam is ending? No. The spam problem is changing. It’s changed before, and it’s changing again. It’s adapting to our efforts. Much is changing to lower volumes of higher-return spam. Phishes, malware (identity theft, credential theft, DDOS platforms) etc. We’re seeing marked increases in the amount of spam (often malware and phishes) sent through compromised accounts through real mail servers and so on.
Botnets aren’t dead yet either. Only a few days ago, Festi reversed most of our volume gains over the past 18 months in a days-long paroxysm of fake anti-virus malware spams. Hopefully that’s short-lived, but it does indicate that the potential is still there.
The battle isn’t ending, it’s changing shape. Due to the nature of how it’s changing, I predict rougher times ahead. It won’t be a cracked pc spewing huge quantities of replica spam that’s easily blocked by IP without adverse consequences to anyone, it’s real mail or web servers spewing smaller amounts of malware, that aren’t as easy to deal with with the current tools - you won’t as easily be able to block a specific IP and not worry about accidentally blocking legitimate content. Like real viruses, they’re becoming immune to our “antibiotics” and acting differently.
Not to mention ongoing attempts to pollute other Internet services such as Facebook or Twitter.
We’re going to have to keep working hard to stay ahead, and not read too much into the absolute volumes of spam emails. The volume is lower. But what’s coming out is much more dangerous.
Ending? Terry, we’ll talk about that over a beer in 5 years ;-)
Heh. Terry - bring Bill along for those beers. He predicted a similar end some years ago.
I said that spam is down, but the abuse problem is not over. It’s simply shifting (agree completely with Chris) from “send tons of garbage in email” to diversifying into other things that Chris spoke about, and they are more difficult to block.
The main reason I commented at all is exemplified by your very last comment: “Spam is down, but the abuse problem is not over”. It’s as if you’re saying that the spam problem is ending, but other forms of abuse aren’t. They’re not other forms of abuse, they’re _still_ spam… It’s almost like the old days, when spam went by
open SMTP relays. When we started to make headway in stopping that, some people did say “spam was defeated”, but it wasn’t, it was just migrating to open web proxies.
They conflated the mechanism with the abuse itself. Killing open relay didn’t kill spam, it just made it mutate the methods it used.
Only this time, to improve return on investment, not only is spam changing methods, the spam is becoming more dangerous too.
Someone reading without full understanding could take your article to mean that email spam is ending, and it’s okay to relax on that front.
That’s the very last message we want to promote.
I’ll rewrite that quote from you in the way I’d like: “Spam is down, but the spam problem is not over, and rougher times are ahead.”
You see the distinction?
I agree.
But you’re missing the fact that not all kinds of spam are the same.
From a broader perspective, what would you rather have leak through your filters? A thousand ordinary replica spams, or a hundred Zeus infectors?
If the former came through easily identifiable botnet zombies, and the latter came through shared real mail servers (ie: Yahoo’s or hotmail’s), which is going to be rougher to deal with? Which battle would you prefer to fight?
The trends are meaningless if you don’t do any further analysis of how the problem is changing.
If you’re in a real battle, and the observer says the number of enemy troops entering the field dropped by a factor of 10, you’d better find out whether the new ones are in tanks, before assuming you’re winning.
You have to have a good idea of what the remaining spam is, and how you’re going to fight it.
For a server provisioning engineer, dropping spam volumes are good. May not have to buy more machines for example. But from a broader health-of-the-environment perspective (ie: customers being defrauded, having their identity stolen, losing confidence in the Internet), you have to know the character of the remaining spam, and you also have to take into account what you’re going to need to stop it.
Absolute volumes are dropping. We all agree on that. That’s great.
What’s not so great is that the contents of the spam are increasingly malware (identity stealers, keyloggers, zeus infectors etc). An end-user falling for one of those has far worse consequences for the health-of-the-environment (not to mention the poor end-user) than spending a few bucks on a fake Rolex, which might never come.
What’s also not so great is that the methods of sending them (cracked accounts on real servers, VPS tunnels to shared hosters bought with stolen credit cards etc) is becoming far more prevalent. What this means for many sites is that they’re going to have to start blocking real servers because they got broken into, but the innocent users trying to use those real servers get blocked too.
In other words, damage to innocent third parties is going to go up.
So, every spam is becoming on average more dangerous, and stopping it will cause more damage to innocent third parties than it has heretofore, because every spam emitter is “holding hostages”. That’s my definition of rougher. Dealing with huge bot armies is a lot easier than the fight we’re now beginning to see take shape.
I’m not, I just didn’t mention it in my original post. I am planning on doing a follow up discussing what you’ve mentioned. I agree with your points that campaigns are narrower and more targeted, and that they are more difficult to detect and remove because they are more integrated into their hosts.
Agreed.