Combating spoofing

Three years ago, I wrote a blog post entitled Combating Phishing talking about what Exchange Online Protection (EOP) does to stop phishing messages [1].

Last year, I wrote one of my most popular blog posts entitled Why does spam and phishing get through Office 365, and what can be done about it?

Recently, I wrote another blog post about how to block phishing when the From: address is the same as the To: address, making it look like an internal message in (Not) Using the Additional Spam Filtering option for SPF hard fail to block apparently internal email spoofing. This blog post recommends customers set up DMARC records of p=none and an Exchange Transport Rule to mark messages as spam if they fail DMARC.

The last item (setting up a DMARC record) is the best way to stop spoofing of your domain. If you add all of your IPs to your SPF record – including any inbound connectors, or permitted 3rd parties who send email as you – everything will work fine. Some of our customers set up the DMARC record but don’t include all 3rd parties in their SPF record, but instead add these 3rd party IPs to IP Allow lists (or to Exchange Transport Rules that operate on the connecting IPs) to skip filtering. This second method is a way of blocking all mail from the outside world as your domain but secretly punching holes for your friends. While I recommend using SPF records to keep track of your friends, the IP Allow route works, too.

However, here’s the problem (it is a problem that is not unique to our customers, it is one that exists everywhere in the industry): not enough people do this. Some set up DMARC records, and some keep track of their 3rd party senders, but many do not. And, spammers keep phishing their domains where the sender (domain in the header.from) looks like an internal message but really is coming from the outside. Without the action of setting up the required DMARC records, there isn’t enough protection. There is a gap between recommendation and implementation.

So how do we fix it?

Here’s how.

You may have noticed a change recently and it can be summarized like this:

If a message arrives from the Internet “from” your domain and destined to your domain, you will notice that unless it authenticates, it will be marked as spam.

Whether or not you publish SPF, DKIM, or DMARC records, any external message that is sent to you as you that doesn’t pass authentication will be marked as spam with the following exceptions:

1.
We try really, really hard to authenticate the sender.
SPF, DKIM, and DMARC are ways that domains can explicitly permit 3rd parties to send email as them. However, what we see lots of evidence for is senders setting up mail servers and transmitting email to themselves from IPs outside of their SPF record.

This happens a lot.  
  
We do some extra DNS lookups of the sender and receiver to determine if we can do what I call “extended authentication.” If a message passes extended authentication, we suppress the spoofing failure.  
  

2. We consider sender reputation as part of our spoofing detection logic.
   There are many 3rd party services that send email as the recipient. Sometimes these services authenticate, sometimes they do not. For example, we ask all of our customers to add “include:spf.protection.outlook.com” to their SPF records. Most of our customers do, but some don’t. If email comes from our IP range, since we know how we route email we can factor that into our antispoofing check.

   There are a lot of sender reputation variables that we use, but the bottom line is that sender reputation may sometimes override a spoofing failure.

3. We use the recipient’s own traffic statistics when determining how to apply the antispoofing logic.
   Sender reputation is one thing, and authentication is another, but recipient reputation is yet another factor. Are they a large customer that gets lots of spoofed messages? Or a small receiver with few? If they get a lot of spoofed messages, how often do they authenticate? Lots or a little? From which senders?

   These factors, plus others, play into the variables that we use to enforce antispoofing logic.

Combining all of these inputs, I can’t give a straightforward “This is how the algorithm works” explanation because the explanation is “It depends on several things” and the rules are not static.

We understand that spoofing is a problem and not everyone understands how to set up SPF, DKIM, and DMARC. But (almost) everyone does understand that unauthorized spoofed messages cause real damage. And that’s who this is supposed to help.

This has the potential to cause false positives. We see plenty of instances where somebody stands up a mail server somewhere from an IP with no reverse DNS record, and sends mail as the domain to the domain, and to no one else. Is that malicious spoofing? Or not?

So, to fix that, we’re back to my recommendations at the start of the post – IPs that send you email as you legitimately should either be added to your SPF record, or barring that as IP Allow rules in the Exchange Admin Center; or, if you own the IPs and they are internal to your organization, added to an inbound on-premise connector (which treats the message sent from them as an internal message that bypasses spam scanning). The more you inventory your IPs, the better the feature works because it will catch more spam and cause fewer false positives.

But even when you don’t, we still try hard to have you covered.

For example, suppose that a domain has many different senders as them from 3rd parties: HR notifications, surveys, automated tools, and bulk senders. Some of these senders are explicitly authorized using SPF or DKIM, some of them are Best Guess Authenticated by EOP internally, and some are added to the domain’s IP allow list or ETR allow. Some other senders are not authorized at all but they are sending legitimate email. Finally, we have the malicious spoofers.

For the above, we have antispoofing protection but because the proportion of unauthenticated email is so high, the protection is less aggressive.

As a domain improves its authentication story using any of the above workarounds, the antispoofing protection gets better.

Whereas the maliciously spoofed case in the first diagram may be (but not necessarily) mistaken for a legitimate but unauthenticated message and get subsequently delivered, in the second case it will be marked as spam. The odds that a legitimate by unauthenticated message getting mistakenly marked as spam are also reduced.

The probability that a spoofed message is legitimate is much smaller so the system can be more confident of marking a message as a malicious spoof. The filters are also self-learning – the more email authenticates, the better antispoofing works. It is a self-correcting mechanism.

[1] That post is out of date.