DMARC one year later, and what have we learned?
It has been one year since I posted that Office 365 now supports inbound DMARC verification. What do we see in terms of how much mail it blocks in production?
Well, we’ve learned a lot of things; some of it good, and some of it bad. I took a look at our network-wide statistics yesterday and a total of 350 million messages arrived at our system with a DMARC record in the 5322.From address (I don’t have numbers on total email). Of that, about 3/4 of the messages passed DMARC and 1/4 of it failed:
That’s a lot of messages failing DMARC, but how much of that is marked as spam?
In Office 365, we don’t reject messages that fail DMARC with p=reject because (a) a lot of our customers have complex routing where they have other services in front of ours that break SPF, DKIM, and DMARC on legitimate email, and (b) a lot of senders don’t have all of their ducks in a row when it comes to SPF, DKIM, and DMARC; and (c) there is still some forwarded email (e.g., mailing lists) that fail DMARC that our customers could not get if we respected DMARC rejects.
So, given that, we override DMARC p=reject and stamp action=oreject for a DMARC fail/p=reject policy, or opctreject if it failed DMARC but the percent=xx randomly overrode a DMARC enforcement (the same for quarantine). Given that, how does the distribution of DMARC fail look like?
You can see that the majority of messages have a “don’t do anything special if this message fails DMARC” policy. Roughly one out of five messages that fail DMARC could get junked.
What do we do with messages that fail DMARC and subsequently go through the filter? Do they all get delivered to the inbox?
No. As it turns out, about 9 in 10 get delivered eventually. Other parts of our filter may still mark them as spam, so even if they fail DMARC with p=none, there is still a chance the message will be junked.
Of that 11%, what sends it to the spam folder? If not the DMARC policy, then what?
And of the non-spam that fails, does our filter also agree that the message is not-spam? Or our customers saying “We don’t care if it fails, it’s a good message so don’t bother even trying to filter it” ?
Here it is below:
Of messages that fail DMARC with p=none, our content filter still marks a message as spam for the majority of those cases. However, sometimes an end user’s blocked sender list (6%) flags it as spam, sometimes a customer’s Exchange Transport Rule (ETR) marks it as spam (4%, no doubt due to this blog post I wrote), and other times we think it’s malware (1%).
The picture is more complex for messages that are marked as non-spam that fail DMARC with p=none. That happens to 89% of DMARC fail/p=none messages, but you can see that much of the time end users are overriding with a safe sender (5%), or the message is an internal message that routes out and back in again within their organization (20%) or they have an IP allow or ETR allow rule (20%). Our filter only gets a chance to make a decision in just over half the cases.
So, clearly, there is a large justification for people to override a DMARC fail action if it happens so often. You may say “Oh, it’s not required because the filter won’t mark it as spam anyway” but in fact these are all pre-emptive overrides just in case because the message is failing sender authentication.
And to prove it, let’s go back to the messages that fail DMARC with p=reject. These should be marked as spam 100% of the time.
But they are not, here’s what that looks like:
In fully one quarter (!) of the cases, a DMARC fail/p=reject is not being marked as spam.
How can this be? Why aren’t 100% of messages that fail DMARC being marked as spam? Don’t we know that all sorts of bad things can happen?
Here’s why these messages are “rescued” from filtering:
From the above:
6% of the messages are internal, meaning that someone has a hybrid mail configuration set up in Exchange Online Protection with the proper connectors between the cloud and their on-prem environment, but these messages are failing DMARC. However, since they treated as internal messages, filtering is not enforced. This is a good thing.
27% (which sounds like a lot) is due to an IP or ETR allow. People know that sometimes a message will fail DMARC but yet they still need to override the verdict because it is a known breakage (e.g., mailing lists, or external senders to themselves that aren’t in their SPF or DKIM records). That part is a good thing.
However, customers also do ETR allow rules just on a domain match. This is not a good thing because it’s easy to spoof a domain. I don’t have the breakdown on that.
6% of the messages are rescued because of end user safe senders. This is a mixed bag. End users want to receive email from legitimate senders who get breakage due to forwarding or routing. But sometimes end user safe sender spammers, too. This is a mixed good/bad thing.
Finally, 61% of messages in this category are not marked as spam because our filters said not to mark it as spam. This is because we know that a lot of our customers have complex routing where SPF, DKIM, and DMARC are broken before we see the email message. If we enforced DMARC reject they would get far too many false positives. In this case we have to suppress DMARC enforcement.
This completely justifies our reticence to enforce DMARC rejects at the connection level. Our customers have complex configurations, there are a lot of broken senders out there, and end users sometimes want to receive messages even if the message fails DMARC.
So in the end, out of 350 million messages that failed DMARC, about 3.5 million were marked as spam because of a DMARC reject record and nothing else – about 1%.
I know that doesn’t sound like a lot, but 3.5 million spoofing/phishing messages that might otherwise have gone undetected is pretty good.