|
||
|
||
URL shorteners, like bit.ly, moby.to and tinyurl.com, do three things:
1. Make a URL shorter
2. Track clicks on the URL
3. Hide the destination URL
Making URLs shorter was their original role, and it’s why they’re so common in media where the raw URL is visible to the recipient—instant messaging, twitter and other microblogs, and in plain text email where the “real” URL won’t fit on a single line.
From the moment they were invented they’ve been used to trick people to click on links to pages they’d rather not visit, from musical classics to less tasteful content. And, in just the same way, spammers quickly found that they were a good way to avoid content-based filters or to hide a suspicious looking target URL.
Inevitably, URL shorteners that are persistently abused by spammers (especially those where that’s done with the support of the URL shortener operator) start to be seen as a sign of spam, and email that uses them will be treated with suspicion by content-based spam filters and often sent to the spam folder.
bit.ly is probably the highest profile URL shortener, so it’s the one you’ll most likely see people trying to use in email. What effects does that have?
Now being “totally owned” by the Canadian Pharmacy gang, thousands of URLs being spammed with very slow takedowns.Not good. —SpamHaus on bit.ly
bit.ly have been on SpamHaus’s radar for quite a while. They’re listed on the SBL multiple times. They’re listed in the DBL—SpamHaus’s newish domain based blacklist, intended for content-based filtering of email. All this means that emails that contain bit.ly URLs are increasingly likely to have serious delivery problems.
This isn’t unique to bit.ly: many other URL shorteners have similar problems—j.mp, su.pr, and others. Nor is it unique to SpamHaus: many other spam filters, public and private, are starting to treat common URL shorteners with suspicion.
Naive use of URL shorteners in your email will send it to the spam folder.
Sponsored byCSC
Sponsored byVerisign
Sponsored byDNIB.com
Sponsored byRadix
Sponsored byIPv4.Global
Sponsored byWhoisXML API
Sponsored byVerisign
A World-Renowned Source for Internet Developments. Serving Since 2002.
Thanks for the additional information, Steve. I’ve been involved in anti-spam/anti-UCE application development for some time and post quite often about the topic on my blog, but your info on URL shortening services was an aspect I hadn’t considered until now.
Services like Bit.ly are so pervasive at this point, thanks largely to the 140-character limit of Twitter, that spam filtering applications may need to perform checks and comparisons to the remainder of the email in an attempt to decide if the shortened URL is intended as spam or simply a valid shortened URL. Yet another hurdle in the analysis of spam content.
Thanks,
Mike Dailey
http://www.daileymuse.com
One reason that Spamhaus lists bit.ly on their DBL is because they are seen in so much spam. However, they are not listed in DBL’s “block” zone but in their “URL shortener” zone. Their own documentation says that you shouldn’t use that zone to block outright, you should use as a weight in the spam filter.
But even then, using bit.ly as a weight in a content filter will be prone to false positives. The vast majority of links in bit.ly are legitimate. It is true that bit.ly is abused and that there are URL shorteners that either are set up for spamming, or don’t do a good job of abuse mitigation, but bit.ly is not among them. They fight abuse; this is straight off their blog:
The first [line of defense in bit.ly abuse prevention] is VeriSign’s iDefense IP reputation service. The iDefense system is focused on detecting and defeating malware. The iDefense blacklist includes URLs, domains, and IP addresses which host exploits, malicious code, command and control servers, drop sites and other nefarious activity.
The second is the Websense Threatseeker Cloud service, which we’ll be adding to our arsenal of anti-spam tools. Websense will analyze the web content behind bit.ly links in real time, using heuristic tools and reputation data to flag spammy URLs, malicious content and phishing sites.
The third is Sophos, an innovative security service whose behavioral-analysis technology goes beyond blacklists, to proactively detect spam and malware.
Obviously, bit.ly cares about making sure that spammers don’t abuse their service. They are not the lazy, fly-by-night single-coder type operation that sets up a redirector and doesn’t notice when someone takes advantage of them.
Because of this, a spam filter that decides to block messages with links to bit.ly will be prone to false positives – lots of them. Bit.ly is the most popular URL shortener. That’s reality and if you block it, users will complain (especially if you have a global antispam business) and it is not worth the support costs. Getting users to change their behavior is asking too much because they are accustomed to seeing and using bit.ly in Twitter.
Blocking mail because it contains a bit.ly link is like the current TSA screening procedures – it’s more trouble than what it is worth. It’s a filtering shortcut, but not a good one.
Sean C. was nice enough to share a Spamassassin plugin that will decode short URLs and verify the real URLs against several URIBLs.
http://www.fsl.com/support/DecodeShortURLs.pm
http://www.fsl.com/support/DecodeShortURLs.cf
-Jim P.
You wouldn’t need URL-shorteners quite as much if sites such as blogs and news sites didn’t make their article URLs so darn long… they feel compelled to cram keywords into the URL until they’re often longer than 80 characters and won’t fit on an RFC-compliant e-mail line.
Twitter needs to raise their character limit to something usable I guess. 500 say. That'd at least make the text there more literate and not full of u, wat, rly .. besides leaving some space for a typical sized url.