Home / Blogs

IP Address Reputation Primer

Laura Atkins

There has been a lot of recent discussions and questions about reputation, content and delivery of email. I started to answer some of them, and then realized there weren't any basic reference documents I could refer to when explaining the interaction. So I decided to write some.

This post is about IP address reputation with some background on why IPs are so important and why ISPs focus so heavily on the sending IP.

Why IP addresses?

ISPs built reputation around IP addresses because it was one bit of data that malicious senders / spammers couldn't forge. The connecting IP is a fundamental part of the network transaction and if you forge an IP then SMTP can't work. Because that was the reliable data they had to work with, that's what they used. Even now, when there are other kinds of data, the IP address is still the first thing the receiving MTA sees.

What is IP reputation?

IP reputation can best be summed up as "past performance is an indicator of future results." In other words if recipients responded well to mail from an IP address in the past, then they're likely to respond well to new mail from that IP address.

How is IP reputation measured?

While each spam filtering company and ISP have their own ways of calculating the reputation of an IP address, there are some similarities in what they measure.

  • How many non-existent email addresses is this IP attempting to deliver to?
  • How many abandoned email addresses is this IP attempting to deliver to?
  • How many "known bad" email addresses (spamtraps) is this IP attempting to deliver to?
  • How many recipients complain about receiving this mail?
  • How many recipients complain about not receiving this mail?
  • How respectful of my resources is this IP?
  • Does this IP keep connections open for long periods of time?
  • Does this IP retry deliveries too aggressively?
  • Does this IP stop mailing addresses after receiving a "user unknown" message?
  • Is this IP address configured as if the associated machine was infected by a virus?
  • Is this IP address listed on blocklists we use?
  • That is by no means an exhaustive list of what ISPs measure. If they can measure it they've tried. If the measurement helps them separate spam mail from not-spam mail then they're using it.

How fast does IP reputation change?

IP reputation is often measured over multiple time periods. ISPs can look at a 1 day, 7 day, 30 day and 90 day reputation. A good analogy is stock prices. Prices can be very volatile in the short term, but more consistent over the long term. A single bad day, where one or more reputation measurements go bad, may affect delivery that day or the next day but won't damage an overall good reputation. Likewise, a few days of improved mail may not be sufficient to counter months of poor reputation.

How is IP reputation used?

Mail from IPs with a high reputation is accepted faster and at a higher rate than mail from IPs with a lower or unknown reputation. IP reputation can also influence whether mail is delivered to the inbox or the bulk folder.

Key IP Reputation takeaways

  • IP reputation is about how recipients react to mail from that IP. Happy, content recipients turn into good delivery.
  • Brief changes (for good or bad) don't necessarily ruin delivery over the long term.
  • Steady improvements will result in improved reputation.
  • It may takes as much time to change a reputation in one direction or another as it took to establish the reputation in the first place.
By Laura Atkins, Founding partner of anti-spam consultancy & software firm Word to the Wise
Follow CircleID on
Related topics: Email, IP Addressing, Spam
SHARE THIS POST

If you are pressed for time ...

... this is for you. More and more professionals are choosing to publish critical posts on CircleID from all corners of the Internet industry. If you find it hard to keep up daily, consider subscribing to our weekly digest. We will provide you a convenient summary report once a week sent directly to your inbox. It's a quick and easy read.

I make a point of reading CircleID. There is no getting around the utility of knowing what thoughtful people are thinking and saying about our industry.

Vinton Cerf, Co-designer of the TCP/IP Protocols & the Architecture of the Internet

Share your comments

Doomed David Conrad  –  Jan 26, 2012 10:23 PM PST

While I understand and sympathize with the rationales and justifications used to make decisions based on the reputation of IP addresses, this strategy is fundamentally doomed.  With IPv6, the vast amount of address space and the ease by which miscreants can change addresses means attempts to use IPv6 addresses as keys in a reputation database are going to get a bit complicated.  What's the point of blocking an IP address if the bad guy can trivially source their spam from 2^64 (or 2^96 or more) -1 alternatives? Worse, with the likely inevitable deployment of large scale/carrier grade NAT, entire universes of users are going to be represented by a single IPv4 address.  It is unlikely that a service that relies upon IP reputation is going to have a happy experience blocking a single IPv4 address when blocking that address is going to block thousands (or more) innocent users/sites/etc.

IP(v4) address reputation was a valuable tool, but that tool's usefulness is rapidly drawing to a close.  Continued use of that tool past its sell-by date is a recipe for unhappy users and a broken Internet.  Folks who currently use IP reputation really need to find alternatives to determining the legitimacy of traffic sources.

Reputation is on the rise The Famous Brett Watson  –  Jan 27, 2012 4:01 AM PST

There's a lot of rhetoric in your comment (the second paragraph in particular), but not a lot of reasoning to back it up. Concentrating on the reasoning alone, I see two points: the vastness of the IPv6 address space, and IPv4 address sharing as a consequence of NAT.

On the first point, yes, it will be ineffective to blacklist individual IPv6 addresses in the face of an opponent with control over his client IP addresses. This makes it unlikely that anyone will try to represent reputation at such a fine-grained level. Instead, the probable approach will be some form of address aggregation, with a reputation assigned to that range of addresses. This already happens with IPv4 anyhow: if numerous addresses in a range exhibit bad behaviour, it's often simplest to describe the whole surrounding address range as disreputable.

The second point, of NAT, is even less of an issue. It has been out there in the wild for some time now, and if it's creating headaches for IP reputation measurement, I haven't heard about it. Your primary concern in this area seems to be the issue of collateral damage — the blocking of "innocent users/sites/etc" along with the bad actors. Again, this is not a new issue. The defining moments in that saga happened in 1998, when Paul Vixie's RBL decided to list the likes of netcom.com and msn.com for being too spam-friendly. The collateral damage was significant at the time, but there was a long-term benefit: network owners have been taught that they can not simply deny all responsibility for the actions of their users — and nobody is too big to block. I believe the Internet today would be a worse place if not for this piece of history.

In short, if ISPs want the reputation of their NAT IP ranges to be positive, it's up to them to enforce the appropriate terms of service on their users. If address-sharing like this makes the job harder, and makes the potential collateral damage worse in the case of a bad reputation, then guess what: that's the network owner's problem, not everyone else's problem.

Reputation, as a concept, is only going to get more important, not less. The changing identity landscape of IPv6 will have its impact, so be sure, but unless you fix human nature, or eliminate all the bad actors from the network, public-facing services can be made much more robust with the appropriate use of reputation information than without.

/48 or /64 is likely to be the new /32 Suresh Ramasubramanian  –  Jan 27, 2012 8:18 AM PST

Given that most tunneled v6 providers are glad to give you very "large sounding" v6 CIDRs - all of them on a tunnel mapped to your 1U pizzabox colo with a single v4 address .. why should reputation providers restrict themselves to blocking /128 at a time?

And v4 is going to be there almost forever at least for email.

Rhetoric? Not so much. Just predictions. David Conrad  –  Jan 27, 2012 9:56 AM PST

The Famous Brett Watson,

Folks who are relying on IP reputation are basing decisions about whether to (for example) accept email or block access to websites based on past traffic emanating from that IP address (or block of IP addresses if you prefer).  In a world where addressing is relatively stable and usually maps one-to-one, this makes sense.  I believe we are rapidly leaving that world.

In the case of IPv6, the (presumably) increasing ease in which one may obtain IPv6 addresses will probably mean it'll be straightforward to get a block, use it, discard it, and get another. Are folks relying on IP reputation going to block address blocks they see for the first time? I hope not.  As for blocking aggregates, how much of a prefix will get blocked?  A /64?  /56?  /48?  /32? How big do you think these block lists will get before they become unmanageable?  Then there is http://www.circleid.com/posts/why_dns_blacklists_dont_work_for_ipv6_networks/.

With respect to IPv4, no longer will single organizations be behind an IPv4 address, rather it'll be a number of (likely dynamically changing) unrelated organizations who happen to map into the same NAT'd address(es).  If a customer of a CGN-deploying major ISP (and I suspect it'll be major ISPs that deploy CGN due to cost) happens to have a machine zombified and subsequently sends out spam, the IP address that spam will be coming from will the the same one shared by (dozens,hundreds,thousands) of other unrelated organizations.  Will reputation services block the entire major ISP? And yes, I can assure you that this is indeed a problem today any place multiple sites get aggregated into a single address.  For example, in multi-site web hosting, the various "cloud"-based services, and shared addressing virtualized server farms.

And then there is the issue of IPv4 address reuse.  Even today with relatively low IPv4 address churn, folks getting 'previously loved' IPv4 address blocks are often faced with trying to get their blocks removed from various blacklists that the previous owners of the blocks happened to have gotten themselves on.  In the future, as address blocks get "transferred" more frequently, this problem is probably going to get (much) worse.

At some point, I figure folks who are relying on IP reputation as a means of blocking are going to get weary of innocent folks calling them up and yelling at them.  I agree reputation as a concept is going to get more important, but basing that reputation on the likely increasingly ephemeral nature of the then current mapping of IP address to traffic generation seems to be a losing proposition.

Rumours of the death of DNSBLs have been greatly exaggerated The Famous Brett Watson  –  Jan 27, 2012 7:43 PM PST

In the case of IPv6, the (presumably) increasing ease in which one may obtain IPv6 addresses will probably mean it'll be straightforward to get a block, use it, discard it, and get another.

It would be unwise to increase the ease with which one may obtain an address block for this very reason. There must be appropriate and reasonable barriers to obtaining additional address blocks, just so that miscreants don't treat it like a cookie jar into which they may dip their hands as they please. Even the IPv6 address space can be exhausted if you treat it as a consumable (rather than durable) resource.

The general problem of reputation laundering through identity change is also a reason why "no reputation" must be distinguished from "good reputation" and "bad reputation".

As for blocking aggregates, how much of a prefix will get blocked?

Suresh has voiced an opinion on this, above. I expect that it will relate to the size of the allocation and the type of network owner. Experience will be necessary to determine what is effective.

As for the article by John Levine to which you provide a link, yes, existing practices in relation to IPv4 don't map straightforwardly into the larger IPv6 space. There is nothing to suggest that this problem is insoluble, however: the article mentions several avenues of research, and I'm sure it's not an exhaustive list. Rumours of the death of DNSBLs have been greatly exaggerated, although it may be time to stop using DNS, as such. DNS was never the ideal protocol, just an adequate and extremely convenient hack.

If a customer of a CGN-deploying major ISP (and I suspect it'll be major ISPs that deploy CGN due to cost) happens to have a machine zombified and subsequently sends out spam, the IP address that spam will be coming from will the the same one shared by (dozens,hundreds,thousands) of other unrelated organizations.  Will reputation services block the entire major ISP?

Whatever address range is used by the NAT will attract a deservedly poor reputation. This may result in the address range being blocked outright, or merely downgraded in various ways. Yes, this is happening even as we speak. It's a poor idea to set up SMTP clients behind a NAT range for this reason (among others). SMTP servers require a static IP address; SMTP clients don't, but it's advisable unless you are a miscreant deliberately trying to work around the effects of a bad reputation.

We don't need NAT to experience this problem: if customers use their ISP's mail servers for outgoing mail, then they share the reputation of those mail servers. This is why it's important for ISPs to be pro-active about acceptable use, including unwitting violations like zombies. This also goes for cloud services and similar arrangements: the service providers must manage the behaviour of their users, or the reputation of the service provider will suffer, and their future customers will be unhappy customers. As I said before, that's the network owner's problem, not everyone else's problem.

And then there is the issue of IPv4 address reuse.

Yes, it's important to investigate the reputation of any address block one might acquire. The same applies for domain names, which might be similarly tarnished by previous owners. Address reuse should be considerably less of an issue under IPv6, though, shouldn't it? So the rise of IPv6 is not all bad news for reputation.

At some point, I figure folks who are relying on IP reputation as a means of blocking are going to get weary of innocent folks calling them up and yelling at them.

No, we've already been down this path. The smart ESPs listen to their customers, not the people sending mail to their customers. So long as the customers are happy, it's all good. The mail senders can be politely instructed to fix the problem at their end, no matter how indignantly they yell. If the customers are complaining about false positives, then there may be a need to do something, but even the customers tend to get the picture when given the option to turn filtering off. Reputation-based filtering doesn't need to be perfect — it just needs to be better when it's turned on than turned off.

There are a couple of lingering issues that I feel are implied by your comments, which need to be addressed explicitly. First, you seem to imply or assume that "reputation is forever"; second, you seem to imply or assume that "bad reputation means outright blocking". Both of these may be true for the simplest approach to the problem: dumping address ranges into a local firewall or "deny service" list. This does happen, but not usually in large scale operations. Reputation service providers (e.g. DNSBLs) treat reputation as a dynamic thing: if an address range cleans up its act, its reputation improves. Similarly, outright blocking is just one possible response to a bad reputation, and not always the best one. There's a lot to be said for accepting delivery of disreputable mail to a "spam" folder.

I do not subscribe to your prophecies of doom.

cassandra IN CNAME drc David Conrad  –  Jan 28, 2012 6:12 PM PST

The Famous Brett Watson,

It would be unwise to increase the ease with which one may obtain an address block for this very reason.

My observation of the RIR policy processes related to the allocation of IPv6 address space is that getting IPv6 addresses has been getting easier over time. It may be that this trend will cease in the future, but my impression is that the way policies are set would suggest otherwise. Further, to quote Suresh:

Given that most tunneled v6 providers are glad to give you very "large sounding" v6 CIDRs - all of them on a tunnel mapped to your 1U pizzabox colo with a single v4 address.

I'm not sure this supports your point regarding not increasing the ease by which one can obtain IPv6 addresses.

With respect to IPv4, I don't think you're taking into account the full implications of the exhaustion of the IPv4 space.  For example, you state:

It's a poor idea to set up SMTP clients behind a NAT range for this reason (among others).

This implies you have a choice.  In a world where IPv4 addresses are either unavailable or are available only at high cost, I'm guessing choices will be limited. It seems likely you will see a proliferation of services (SMTP included) residing behind CGN/LSN. The obvious alternatives, such as relying on "cloud" based services, are undoubtedly going to proliferate, but I seem to recall a non-trivial amount of spam (etc) originating from those services today and I'm unsure how effective reputation services are against Google, AWS, etc.  Maybe this will improve in the future.

I believe that reputation services that base decisions on increasingly ephemeral data such as the IP address from which badness originates are likely to cause more and more non-trivial collateral damage over time.  It may be that you are correct and people will ignore the damage and continue to use such services (I have some skepticism since communication between two parties tends to require both parties are able to hear each other), but I would argue that this does not bode well for the reliability of Internet services or for the Internet's model of providing those services.

I do not subscribe to your prophecies of doom.

The nice thing about making predictions is that all you have to do is wait…

Carrier grade NAT is already a "known feature" on cellular carriers, a lot of broadband ISPs .. Suresh Ramasubramanian  –  Feb 01, 2012 2:23 AM PST

But you will find few if any actual mailservers running behind one of those. You will certainly find a lot of email clients (outlook etc on the laptops of people using 3G modems) as well as botted PCs emitting virus generated spam and other malicious traffic

These are hardly the sort of IPs you would expect to connect directly to your MX to deliver mail inbound to you.  Outbound mail on the other hand - yes, certainly - but I can't think of any ISP dumb enough to use a DNSBL to blindly filter their outbound mail stream.

So I would put it to you that the IP address of a real mailserver is just not likely to be ephemeral, whereas dhcpv6 connected devices - sure.

And I would also put it to you that real mailservers, emitting real SMTP traffic, are going to be a vanishingly small fraction of the larger v6 IP space, compared to whatever other IP capable devices that get v6 connectivity.

No filtering provider relies 100% on IP reputation.  But it is an extremely important data point for them.  And I just don't see v6 suddenly deprecating IP reputation at all.

To post comments, please login or create an account.

Related

Topics

Cybercrime

Sponsored byThreat Intelligence Platform

Cybersecurity

Sponsored byVerisign

New TLDs

Sponsored byAfilias

DNS Security

Sponsored byAfilias

Whois

Sponsored byWhoisXML API

Domain Names

Sponsored byVerisign

IP Addressing

Sponsored byAvenue4 LLC