Troubleshooting the red (Suspicious) Safety Tip for fraud detection checks

Introduction

It has now been about 8 months since we released our antispoofing protection in Office 365, a feature that defends against Business Email Compromise, where the From and To domains are the same. You can read more about that feature at my other blog post Antispoofing in Office 365. To summarize, it defends against others spoofing your domain in the From: address - the one that appears in your email client - by figuring out if the sender is legitimate or malicious. It's similar to how DMARC works, the key difference is that while DMARC looks for a DNS record published at _dmarc.<domain>, the antispoofing protection interpolates if it does not exist. That is, if the domain in the From: address does not publish a DMARC record, what would it say if it did?

It has also been a few months since we first started rolling out Safety Tips to Office 365 customers (everyone will get them by mid-November). Since that time, the question that has arisen the most is this: Why am I getting a red Safety Tip in my email? This is a legitimate sender!

Note: the above is inserted into the HTML part of a message. If there is no HTML, the text is added as plain-text.

For the search engines, that's "This sender failed our fraud detection checks and may not be who they appear to be." .

Figuring out the base conditions of why it was stamped

This red safety tip gets stamped when a message fails a spoofing check. Domains that are provisioned with an organization are spoofed are known as self-to-self spoofing (or intra-org spoofing), and is applied to all customers within Office 365. Domains that are external to an organization (e.g., google.com) are known as cross-domain (or external domain) spoofing. This is only available to Advanced Threat Protection and E5 customers of Office 365.

There are 2 common reasons why a "legitimate" message gets a red Safety Tip about a message failing fraud detection checks when the From: domain and To: domain are the same, and 1 less common reason:
.

  1. The sender is sending emails from an unauthorized source that is sending as your domain (all customers), or an external domain (ATP customers) in the SMTP MAIL FROM, but not in the domain's SPF record, and is also not signing the emails with DKIM
    .
    This occurs when a department within your organization or the sending domain's organization spins up a mail server, or outsources to a Software-as-a-Service provider, who sends email as your domain but they are not in the SPF record. To determine this, open up the message headers and look at the Authentication-Results header. Suppose the service that is sending emails as you (or in the example below, contoso.com) is HRnotifications.com and they are sending from the IP 1.2.3.4:

     Authentication-Results: spf=fail (sender IP is 1.2.3.4)
      smtp.mailfrom=contoso.com; dkim=none (message not signed)
      header.d=none; dmarc=none action=none
      header.from=contoso.com; compauth=fail reason=001
    

    In this case, the SMTP MAIL FROM is contoso.com, and the From: domain is also contoso.com. Contoso.com also does not publish a DMARC record. A case like this will almost always get a red Safety Tip and mark as spam because even though a DMARC record is not published, it would have failed had it been.

    I say "almost always" because Office 365 does suppress the Safety Tip and mark as spam action if we have enough reputation data to determine that this is a valid message. It works most of the time, but not for small senders, nor for senders with insufficient reputation, or senders with poor reputation.

    The last tag in the Authentication-Results header is the compauth property. This stands for composite authentication and is Office 365's combined SPF, DKIM, DMARC, and all other internal results. If a message fails explict authentication (DMARC quarantine/reject), the reason code will be 000. If a message does not publish a DMARC record and Office 365 has no valid signals on the message, the reason code will be 001 (other codes are reserved for internal use, but a failed message always starts with 0xx). The compauth result is only stamped for ATP customers.

    Even if the domain does not have an SPF record, the antispoofing check will still apply.

     Authentication-Results: spf=none (sender IP is 1.2.3.4)
      smtp.mailfrom=contoso.com; dkim=none (message not signed)
      header.d=none; dmarc=none action=none
      header.from=example.com; compauth=fail reason=001
    

    .

  2. The sender is sending emails that pass SPF or DKIM, but neither the domain that passes SPF nor the domain the passes DKIM aligns with the From: domain
    .
    Just passing SPF or DKIM is insufficient to suppress the Safety Tip or mark as spam. For example:

     Authentication-Results: spf=pass (sender IP is 1.2.3.4)
      smtp.mailfrom=HRnotifications.com; dkim=pass (signature was valid)
      header.d=HRnotifications.com; dmarc=none action=none 
      header.from=example.com; compauth=fail reason=001
    

    Even though this passed SPF and DKIM, HRnotifications.com != example.com. One of the SPF-passing or DKIM-signing domains must align with example.com. If it doesn't, a Safety Tip will be inserted unless Office 365 determines that the sending IP or domain has enough reputation to suppress the Safety Tip.

    Once again, the compauth has failed with a reason code of 001 (implict auth failure). Self-to-self spoofing does not yet stamp this code, and neither does non-ATP customers in Office 365. However, you can still figure it out by looking at the Authentication-Results (since compauth=fail reason=001 only applies to ATP customers, cross-domain spoofing is not applied to non-ATP customers either).
    .

  3. [In process of being fixed] The sender publishes a DMARC record, but Office 365 had a DNS lookup failure and DMARC TempError'ed
    .
    A corner case we are working on fixing is when a message ought to have passed DMARC, but didn't because of a DNS lookup failure:

     Authentication-Results: spf=pass (sender IP is 1.2.3.4)
      smtp.mailfrom=HRnotifications.com; dkim=pass (signature was valid)
      header.d=notifications.example.com; dmarc=temperror action=none
      header.from=example.com
    

    In this case, DMARC should have passed by aligning on the DKIM domain with the From: domain, but it couldn't because of the DNS lookup failure. A case like this may fail the antispoofing check and insert the red Safety Tip.

    We are working on this to treat it as a DMARC BestGuessPass and supersede the TempError.
    .

Common scenarios when this occurs

When do the first two occur most often?

The IETF has published RFC 7960 - Interoperability Issues between DMARC and Indirection Email Flows. This document describes all the times when a valid message fails DMARC.

Since Office 365's Exact-Domain antispoofing check is similar to DMARC, it is subject to some of the same problems. The difference between regular DMARC and Office 365's antispoofing checks is that our checks will try to figure out automatically that the sender, though failing authentication, is legitimate so you don't have to do anything; whereas DMARC makes you explicitly configure things to make it work. Office 365 doesn't always figure it out automatically so the more you configure it, the lower the chances that legitimate email will get marked as spam.

RFC 7960 was written by some smart people, and have done a great job to inventory as many different scenarios that are possible. The ones that I see the most often are the following, along with their solutions (I discuss this more in https://aka.ms/AntispoofingInOffice365):
.

  1. A third party is sending email, on your domain's behalf, but is not in your SPF record or DKIM-signing as your domain
    .
    This constitutes the majority of cases. Even we here at Microsoft see this all the time. Not a week goes by where someone doesn't ping me asking why they are getting the red Safety Tip. It's because Microsoft has clamped down on spoofing because the problem of exact-domain spear phishing is so prevalent.The fix for this, either:
    .
    a)  Add the sender's IPs to your SPF record, or add the include:<their domain> to your SPF record. See Set up SPF in Office 365.
    .
    OR
    .
    b)
    Have the sender DKIM-sign as your domain. This requires them giving you to publish a public key at a selector you specify, e.g., myselector, and then publishing the key at myselector._domainkey.<yourdomain>.
    .
    OR

    c)  Add the sender's IPs to your IP Allow list, see Configure the Connection Filter Policy in Office 365.
    .
    OR
    .
    d)
    Create an Exchange Transport Rule (ETR) allow rule:- Add the sender's IPs to an ETR Allow rule- Or, if you want to be more secure, add the sender's IPs + the sender domain to an ETR Allow rule
    - Or, if you want to impress your friends, create an ETR Allow rule that looks for "spf=pass" in the Authentication-Results header and the sending domain in the envelope is the domain sending on your behalf (e.g., HRnotifications.com). Alternatively, you look for "dkim=pass (signature was verified) header.d=<signing domain>" in the Authentication-Results header. You would use this ETR in the case that the sending domain passes SPF or DKIM, but is not the same as your domain (i.e., they are sending on your behalf and authenticating, but not aligning)Both option (c) and option (d) will skip filtering and suppress the Safety Tip. End user Safe Senders will skip filtering but will not suppress the Safety Tip.See Transport rules in Office 365.
    .
    OR
    .
    e)
    Use the Set-PhishFilterPolicy to specify senders who are allowed to send to you on your behalf, or an external domain that is sending spoofed email that you think is legitimate. See Set-PhishFilterPolicy. This is only available to Advanced Threat Protection customers.
    .

  2. You are sending to a mailing list which is modifying and replaying the message back into your environment
    .
    This occurs with the following routing path:

     Office 365 -> Mailing list (pass SPF, DKIM, DMARC... then modify the message) -> Office 365
    

    This is one of the hardest ones to solve. Section 4.1.3 of RFC 7960 talks about this. If you are in control of the mailing list, there are a couple of good options you can make to your mail server:

    a) Set it up to not modify the message (not the most popular option)

    b) Rewrite the From: address so that it doesn't fail alignment

    Suppose the message you send looks like this when sending out:

    From: Example User <example@contoso.com>
    To: Mailing List <list@mailingList.org>

    When it goes to the list and gets replayed, it could be rewritten like this:

    From: Example User via Mailing list <list@mailingList.org>
    To: Example User <example@contoso.com>
    Reply-To: Example User <example@contoso.com>

    Not everyone in the email industry likes this solution, but I think it's pretty good if the list server is under your control.

    I talk more about this here: https://blogs.msdn.microsoft.com/tzink/2017/03/22/a-tip-for-mailing-list-operators-to-interoperate-with-dmarc-to-avoid-failures/. It is strongly recommended that you pursue this path when possible.

    c) Submit it as a false positive to Office 365 and we'll see if we can add it to our list of "Do not enforce antispoofing checks from these senders" list. This is a manually maintained list and we only add mailing lists to it, and it takes a long time to update. But if the list is not under your control, this is an option.
    .

  3. You are forwarding email into and out of Office 365 and modifying the message. Sometimes on purpose, but sometimes not.
    If your email path goes like this:

     Internet -> Office 365 -> on-prem mail server -> Office 365
    

    If the on-prem mail server modifies the message, it could fail antispoofing. This occurs if you are running Exchange on-prem which modifies messages (see here for details), or you have another appliance that inserts footers into a message or does some other modification.

    If you are running Exchange, the solution for this is to set up connectors so that the headers we stamp on the first pass through Office 365 are respected when you relay it back into the service. If you are not running Exchange, you will have to set up a few ETRs to respect the original spam or non-spam verdicts and route the message accordingly.

Those are the ones I see the most for customers of Office 365.
.

Sometimes this shows a tip and other times it doesn't!

That's expected. Our backend systems make an attempt to figure out if the sending pattern is legitimate or not. These are dynamic and can change.

For best results, you should authenticate your email and ensure that the authenticated domain aligns with the From: address.

Can I create a Domain Allow (SFV:SKA) or can users create safe senders (SFV:SFE) to suppress the tip?

No, those won't work (or, they may work but not in every case). And you wouldn't want them to, either. Domain Allows are string matches, which means that any spammer can spoof that sending domain or address in the From: address. The message will bypass regular filtering, but it will still get the safety tip to indicate that the sender did not pass fraud detection checks.

If you want to suppress the tip, see the instructions above.

What about disabling Safety Tips?

As a last-ditch attempt to solve the problem, you may be tempted to disable Safety Tips altogether. My advice is this - Don't do it!

Safety Tips are like a seat belt in your car. If it's too tight, or sits too high, or scratches you, the solution is to adjust it or buy one of those thingies that adds padding to it. The solution is not to drive without it.

Safety Tips helps protect you against spoofing, but it also helps defend against phishing (another red safety tip), to tell you when a sender is trusted, when you've skipped filtering due to IP Allows or Safe Senders, and other types of impersonation (coming soon in 2017). So by turning off Safety Tips, you've disabled all that protection.

You are much better off working through the issues about by inventorying IPs, getting proper DKIM signing,  fixing mailing lists, and making sure connectors are set up properly. That ensures you have the best protection possible. It is a little inconvenient (it's inconvenient here at Microsoft, too) but it is worth the effort.

But if you must turn off Safety Tips, you can do that here: Enable or disable Safety Tips in Office 365.
.

Conclusion

Hopefully this article helps you figure out why a message has that red Safety Tip inserted above. We do our best to make sure we only insert it in cases where we think the email is fraudulent, but sometimes legitimate email gets the tip. Fortunately, you can fix it yourself by following the steps above.

If you have questions, just let me know in the comments.