|
||
|
||
There has been a lot of recent discussions and questions about reputation, content and delivery of email. I started to answer some of them, and then realized there weren’t any basic reference documents I could refer to when explaining the interaction. So I decided to write some.
This post is about IP address reputation with some background on why IPs are so important and why ISPs focus so heavily on the sending IP.
Why IP addresses?
ISPs built reputation around IP addresses because it was one bit of data that malicious senders / spammers couldn’t forge. The connecting IP is a fundamental part of the network transaction and if you forge an IP then SMTP can’t work. Because that was the reliable data they had to work with, that’s what they used. Even now, when there are other kinds of data, the IP address is still the first thing the receiving MTA sees.
What is IP reputation?
IP reputation can best be summed up as “past performance is an indicator of future results.” In other words if recipients responded well to mail from an IP address in the past, then they’re likely to respond well to new mail from that IP address.
How is IP reputation measured?
While each spam filtering company and ISP have their own ways of calculating the reputation of an IP address, there are some similarities in what they measure.
How fast does IP reputation change?
IP reputation is often measured over multiple time periods. ISPs can look at a 1 day, 7 day, 30 day and 90 day reputation. A good analogy is stock prices. Prices can be very volatile in the short term, but more consistent over the long term. A single bad day, where one or more reputation measurements go bad, may affect delivery that day or the next day but won’t damage an overall good reputation. Likewise, a few days of improved mail may not be sufficient to counter months of poor reputation.
How is IP reputation used?
Mail from IPs with a high reputation is accepted faster and at a higher rate than mail from IPs with a lower or unknown reputation. IP reputation can also influence whether mail is delivered to the inbox or the bulk folder.
Key IP Reputation takeaways
Sponsored byIPv4.Global
Sponsored byVerisign
Sponsored byCSC
Sponsored byVerisign
Sponsored byDNIB.com
Sponsored byRadix
Sponsored byWhoisXML API
A World-Renowned Source for Internet Developments. Serving Since 2002.
While I understand and sympathize with the rationales and justifications used to make decisions based on the reputation of IP addresses, this strategy is fundamentally doomed. With IPv6, the vast amount of address space and the ease by which miscreants can change addresses means attempts to use IPv6 addresses as keys in a reputation database are going to get a bit complicated. What’s the point of blocking an IP address if the bad guy can trivially source their spam from 2^64 (or 2^96 or more) -1 alternatives? Worse, with the likely inevitable deployment of large scale/carrier grade NAT, entire universes of users are going to be represented by a single IPv4 address. It is unlikely that a service that relies upon IP reputation is going to have a happy experience blocking a single IPv4 address when blocking that address is going to block thousands (or more) innocent users/sites/etc.
IP(v4) address reputation was a valuable tool, but that tool’s usefulness is rapidly drawing to a close. Continued use of that tool past its sell-by date is a recipe for unhappy users and a broken Internet. Folks who currently use IP reputation really need to find alternatives to determining the legitimacy of traffic sources.
There's a lot of rhetoric in your comment (the second paragraph in particular), but not a lot of reasoning to back it up. Concentrating on the reasoning alone, I see two points: the vastness of the IPv6 address space, and IPv4 address sharing as a consequence of NAT. On the first point, yes, it will be ineffective to blacklist individual IPv6 addresses in the face of an opponent with control over his client IP addresses. This makes it unlikely that anyone will try to represent reputation at such a fine-grained level. Instead, the probable approach will be some form of address aggregation, with a reputation assigned to that range of addresses. This already happens with IPv4 anyhow: if numerous addresses in a range exhibit bad behaviour, it's often simplest to describe the whole surrounding address range as disreputable. The second point, of NAT, is even less of an issue. It has been out there in the wild for some time now, and if it's creating headaches for IP reputation measurement, I haven't heard about it. Your primary concern in this area seems to be the issue of collateral damage -- the blocking of "innocent users/sites/etc" along with the bad actors. Again, this is not a new issue. The defining moments in that saga happened in 1998, when Paul Vixie's RBL decided to list the likes of netcom.com and msn.com for being too spam-friendly. The collateral damage was significant at the time, but there was a long-term benefit: network owners have been taught that they can not simply deny all responsibility for the actions of their users -- and nobody is too big to block. I believe the Internet today would be a worse place if not for this piece of history. In short, if ISPs want the reputation of their NAT IP ranges to be positive, it's up to them to enforce the appropriate terms of service on their users. If address-sharing like this makes the job harder, and makes the potential collateral damage worse in the case of a bad reputation, then guess what: that's the network owner's problem, not everyone else's problem. Reputation, as a concept, is only going to get more important, not less. The changing identity landscape of IPv6 will have its impact, so be sure, but unless you fix human nature, or eliminate all the bad actors from the network, public-facing services can be made much more robust with the appropriate use of reputation information than without.
Given that most tunneled v6 providers are glad to give you very "large sounding" v6 CIDRs - all of them on a tunnel mapped to your 1U pizzabox colo with a single v4 address .. why should reputation providers restrict themselves to blocking /128 at a time? And v4 is going to be there almost forever at least for email.
The Famous Brett Watson, Folks who are relying on IP reputation are basing decisions about whether to (for example) accept email or block access to websites based on past traffic emanating from that IP address (or block of IP addresses if you prefer). In a world where addressing is relatively stable and usually maps one-to-one, this makes sense. I believe we are rapidly leaving that world. In the case of IPv6, the (presumably) increasing ease in which one may obtain IPv6 addresses will probably mean it'll be straightforward to get a block, use it, discard it, and get another. Are folks relying on IP reputation going to block address blocks they see for the first time? I hope not. As for blocking aggregates, how much of a prefix will get blocked? A /64? /56? /48? /32? How big do you think these block lists will get before they become unmanageable? Then there is http://www.circleid.com/posts/why_dns_blacklists_dont_work_for_ipv6_networks/. With respect to IPv4, no longer will single organizations be behind an IPv4 address, rather it'll be a number of (likely dynamically changing) unrelated organizations who happen to map into the same NAT'd address(es). If a customer of a CGN-deploying major ISP (and I suspect it'll be major ISPs that deploy CGN due to cost) happens to have a machine zombified and subsequently sends out spam, the IP address that spam will be coming from will the the same one shared by (dozens,hundreds,thousands) of other unrelated organizations. Will reputation services block the entire major ISP? And yes, I can assure you that this is indeed a problem today any place multiple sites get aggregated into a single address. For example, in multi-site web hosting, the various "cloud"-based services, and shared addressing virtualized server farms. And then there is the issue of IPv4 address reuse. Even today with relatively low IPv4 address churn, folks getting 'previously loved' IPv4 address blocks are often faced with trying to get their blocks removed from various blacklists that the previous owners of the blocks happened to have gotten themselves on. In the future, as address blocks get "transferred" more frequently, this problem is probably going to get (much) worse. At some point, I figure folks who are relying on IP reputation as a means of blocking are going to get weary of innocent folks calling them up and yelling at them. I agree reputation as a concept is going to get more important, but basing that reputation on the likely increasingly ephemeral nature of the then current mapping of IP address to traffic generation seems to be a losing proposition.
The Famous Brett Watson,
My observation of the RIR policy processes related to the allocation of IPv6 address space is that getting IPv6 addresses has been getting easier over time. It may be that this trend will cease in the future, but my impression is that the way policies are set would suggest otherwise. Further, to quote Suresh:
I’m not sure this supports your point regarding not increasing the ease by which one can obtain IPv6 addresses.
With respect to IPv4, I don’t think you’re taking into account the full implications of the exhaustion of the IPv4 space. For example, you state:
This implies you have a choice. In a world where IPv4 addresses are either unavailable or are available only at high cost, I’m guessing choices will be limited. It seems likely you will see a proliferation of services (SMTP included) residing behind CGN/LSN. The obvious alternatives, such as relying on “cloud” based services, are undoubtedly going to proliferate, but I seem to recall a non-trivial amount of spam (etc) originating from those services today and I’m unsure how effective reputation services are against Google, AWS, etc. Maybe this will improve in the future.
I believe that reputation services that base decisions on increasingly ephemeral data such as the IP address from which badness originates are likely to cause more and more non-trivial collateral damage over time. It may be that you are correct and people will ignore the damage and continue to use such services (I have some skepticism since communication between two parties tends to require both parties are able to hear each other), but I would argue that this does not bode well for the reliability of Internet services or for the Internet’s model of providing those services.
The nice thing about making predictions is that all you have to do is wait…
But you will find few if any actual mailservers running behind one of those. You will certainly find a lot of email clients (outlook etc on the laptops of people using 3G modems) as well as botted PCs emitting virus generated spam and other malicious traffic These are hardly the sort of IPs you would expect to connect directly to your MX to deliver mail inbound to you. Outbound mail on the other hand - yes, certainly - but I can't think of any ISP dumb enough to use a DNSBL to blindly filter their outbound mail stream. So I would put it to you that the IP address of a real mailserver is just not likely to be ephemeral, whereas dhcpv6 connected devices - sure. And I would also put it to you that real mailservers, emitting real SMTP traffic, are going to be a vanishingly small fraction of the larger v6 IP space, compared to whatever other IP capable devices that get v6 connectivity. No filtering provider relies 100% on IP reputation. But it is an extremely important data point for them. And I just don't see v6 suddenly deprecating IP reputation at all.