Sender authentication part 8: Best-Guess SPF


I've had a document sitting on my shelf (ie, the window-sill 10 feet away from my desk) for about 6 months now just waiting to be read.  It's entitled Sender Repuration in a Large Webmail Service.  It's by Bradley Taylor, at Google, and is available to be read at the past documents from the Conference in Email and Antispam, 2006.

Anyways, I finally got around to reading it this week.  Like everyone else, Gmail uses a lot of Sender Authentication to do their filtering.  One of the ways they authenticate mail is with SPF, and for domains without SPF they use an algorithm they dub "Best-Guess SPF" which is meant to be a temporary measure until more domains come onboard and start publishing their SPF records.  They readily admit that the technique isn't perfect, but it's not bad, either.  Basically, it works in the following manner:

1. Check the domain of the envelope sender.  If it doesn't publish SPF records, then check the MX-records and A-records of the sender's domain.  If the sending domain comes from the same range of IPs as the MX-record or A-record, then the sender has been authenticated.

Example 1 (using fictitious numbers)

Transmitting IP = 4.8.15.16
Envelope sender = me@lost.com
A-record of lost.com = 4.8.15.11
MX-record of lost.com = 4.8.15.0/27 (4.8.15.0 - 4.8.15.31)

Since the transmitting IP is within the range of the MX-records (an abnormally large MX record, but hey, this example is fictitious), we have an authentication.


2. If that doesn't work, get the reverse DNS of the sending IP.  If it matches the domain of the envelope sender, then the sender has been authenticated.

Example 2

Transmitting IP = 4.8.15.16
Envelope sender = me@lost.com
Reverse DNS of 4.8.15.16 = lost.com

The reverse DNS name matches the name of the domain in the envelope sender, so the sender is authenticated.

Example 3

Transmitting IP = 16.23.42.108
Envelope sender = me@others.com
Reverse DNS of 16.23.42.108 = island.com

The reverse DNS name does not match the envelope sender, therefore, no sender authentication.

3. If that doesn't work, use a technique that is referred to as PTR zone.  If the sender is a subdomain of the DNS PTR's zone, then it is authenticated as if the sender comes from the zone itself.  The example given in the document where I discovered this seems a bit backwards, so I'm going to clean it up a bit in order to conform to the description given.


Example 4

Transmitting IP = 16.23.42.108
Envelope sender = me@island.others.com
Reverse DNS of 16.23.42.108 = domain in PTR zone = airplane.others.com

This is close, but not an authentication.  The envelope sender (island.others.com) is not a subdomain of the domain in the PTR zone (airplane.others.com).

Example 5

Transmitting IP = 16.23.42.108
Envelope sender = me@island.others.com
Reverse DNS of 16.23.42.108 = domain in PTR zone = others.com

The domain of the sender (island.others.com) is a subdomain of others.com, and therefore we have an authentication.

Using this extra bit of authentication allows Gmail to authenticate almost twice as much mail as a standard SPF check.  That's actually pretty good.  As to whether or not this is a good idea, OpenSPF has this to say about it:

Best-guess processing is a crude, non-standard attempt at guessing the IP address range of a domain's outgoing mailservers.  "Non-standard" means it is not standardized and specific to the implementation.

...

Some find this remarkably good at detecting unforged messages from domains that have not yet published SPF records. Others consider it a security hole because it gives attackers a lot of additional potential targets (authorized hosts) to hack in order to abuse the domain.

From an anti-spam perspective, I think sender authentication is a good idea but it all depends on how it is used.  In my opinion, successful authentication is best used in conjunction with safe senders.  I first voiced my opinion a few weeks ago when I thought it was a security risk.  However, at the time I don't think I was thinking ahead; I think the way to implement a safelist is to allow a sender to be on a safelist (ie, bypass spam filtering) if the sender can be authenticated. That way, spoofing the sender doesn't work and if they do start spamming, there's a much more reliable paper trail. 

Comments (5)

  1. Al says:

    This is quite insightful…thanks for sharing!! It’s great to see documentation/confirmation that Google’s actually doing something with authentication data.

  2. Norman Diamond says:

    Did you intend to use real IP address this time instead of starting with numbers above 256?  One of them belonged to a famous rogue, a company that actively supported spammers in all their roles, though I don’t know if they still have the same policy after a merger.

  3. tzink says:

    Not exactly.  I picked numbers from the TV show Lost, though I knew that it represented a real IP address.  I then applied fictional results for reverse DNS lookups.

    This represents a departure from my policy of using fictitious IP addresses.

  4. iam@slicknick.com says:

    You guys must be messaging nerds.  Who in their right mind would read this boring stuff?  I did. 🙂  Cleaver techniques I must admit.  

    Can anyone think how SPF would be a security risk?  Another messaging engineer claims SPF is not as safe as people think , but would not comment on what’s the security risk with implementing it and that’s they reason why his company doesn’t use it.   The only thing I can think of is that it can be improperly set, but even that would not be any riskier than not having it.  The most harm you can possibly do is not including all your outbound gateway which would make it too secure, but not any less secure.  Anyway, any opinion would be appreciated.  This is how I bump into this site by googling "security risk with SPF".

    Thanks,

    -Nick

  5. shyamala says:

    Just a clarification

    When sender domain is not a subdomain of PTR Zone you have said sender is not authenticated.What is the right approach in my case below.

    Envelop-Sender is <bluexxcc@mozilla.com>

    Transmitting IP is 63.245.208.141

    Non-authoritative answer:

    141.208.245.63.in-addr.arpa     name = corp01.sj.mozilla.com

    208.245.63.in-addr.arpa nameserver = ns2.mozilla.org

    208.245.63.in-addr.arpa nameserver = ns1.mozilla.org

    mozilla.com is not a subdomain of corp01.sj.mozilla.com.How am I supposed to handle this?

    thanks,

    Krith

Skip to main content