Office 365 and outlook.com/Hotmail are converging infrastructure

If you’ve talked to me in person over the past few months, you may have heard me talk about this. But if not, I’ll talk about it in this blog post and what it means.

Some background

Exchange Online Protection (EOP), the mail filtering branch of Office 365 (I use the terms interchangeably) is the email-for-enterprise part of Microsoft. EOP originally started off as Frontbridge Technologies, based out of Los Angeles. It was acquired by Microsoft in 2005 and was renamed a few times, most recently as Forefront Online Protection for Exchange (FOPE). All of the backend infrastructure of FOPE has been retired and EOP is entirely built on a different platform.

Hotmail was acquired by Microsoft in the 1990’s and a couple of years ago was rechristened as outlook.com, which I will heretofore refer to as “consumer”. It ran on a different filtering stack, and email filtering technology, than EOP/FOPE. We shared some technology but not that much as the email in consumer is different than the email in enterprise. Furthermore, the architectural differences between the two made sharing technologies difficult even if we wanted to share. While we negotiated third party vendor contracts as a company, we each implemented them differently.

Over the past several months, the two teams merged – Hotmail and EOP together at last. That’s all well and good, the two teams are together. But what next?

The way forward

The next step is to merge the two platforms. I would argue that Hotmail’s spam filter technology is more advanced than EOP’s (but not in all aspects) but that EOP’s backend platform is more advanced than Hotmail’s (although not necessarily in all aspects). For example, Hotmail only started supporting TLS in either 2013 or 2014 but FOPE/EOP has supported it for at least a decade. Furthermore, the mailbox infrastructure in EOP is more suitable for redundancy, failover, and message tracing.

So, starting six months ago and continuing into the foreseeable future, Hotmail is moving onto the EOP infrastructure and it is an absolutely massive undertaking. We have to ensure all of the features that we available in Hotmail are supported in EOP – that includes MTA, spam filter, and web UX – and then we have to ensure that EOP is properly equipped to handle a 5x (?) increase in load. EOP’s infrastructure is “heavier” than Hotmail’s because the requirements for enterprise (i.e., redundancy, queuing) are more demanding.

Some challenges

Hotmail is also a different customer than any other tenant in EOP. One of EOP’s strengths is its geolocale feature – customers are provisioned in a particular geo-region. If you’re based in North America, all of your email flows through data centers in North America. If you’re based in Europe, all of your email flows through Europe.

However, Hotmail has a global footprint. That means that a single customer has to span every single geo-region. We don’t have any customers like that currently.

One of the big benefits for Hotmail/outlook.com users is that if they are both users of that service and then for Office 365, they get the same platform and look-and-feel in both. The UX is the same, and they integrate a lot better. You can move from one to the other without having to relearn a bunch of things.

In addition, antispam technologies that work in one can move to the other since the backend is the same. Because the pipes that plug into both are both now hooked up to each other (sort of; it’s not quite this simple) we can leverage technology between the two a lot better. The two services probably won’t use the same filter, but each will have parts in common.

The drawback of migration is that it means we have to devote a lot of time and resources towards migrating users, and this results in opportunity cost of not being able to do new features that can help improve the overall customer experience. Having been through two (!) rewrites already before this, that’s frustrating for me. But in the end, having two services running on different stacks is unsupportable.

What I’m working on that relates to this

Unlike many other members of my team, I am peripherally involved in the migration of Hotmail-to-EOP. The biggest ones I am doing:

1. The Safety UX

   Hotmail has a green shield for senders who are heavily spoofed but authenticate, but Outlook Web Access (OWA) does not. I’m working on overhauling the safety experience in OWA so that users can know why a message is marked up the way it is, and then carry over this experience to enterprise and not only in consumer.

2. Backscatter protection with Boomerang

   You’ve heard me talk about Backscatter protection with Boomerang before. We just released it enterprise, and it’s a necessary component before migrating consumer mailboxes.

3. DKIM and DMARC
   EOP already supports inbound DKIM verification, but so does outlook.com. EOP supports DMARC and so does outlook.com, but EOP does it a little differently (I have other blog posts that explain the difference). However, EOP has yet to turn on DMARC reports, we have to fix a couple of items in the Exchange MTA before we do. Then we will be at parity with Hotmail.

   But even more than that, EOP is currently working on DKIM-signing (it’s on our public roadmap). Eventually, when Hotmail moves over to EOP I’d like to get it to start signing with DKIM. That will be an improvement to the Hotmail service, rather than a simple migration.

   People sometimes ask me “Does Hotmail plan to create a DMARC record with p=reject the same way that Yahoo and AOL do?”

   My response is non-committal. Before we can look at that, we need to get the fix in Exchange completed to preserve message content. We can deploy this fix in Office 365, but we also need to push it to various versions of on-premise Exchange server; what versions do we push it to? How much uptake is enough? Next, we need to sign all mail in Hotmail with DKIM which means we have to migrate the entire user base of Hotmail over to EOP. Then, we have to measure and assess the potential impact.

   So before I can even answer the question, I need to know what will happen if we do and then do a cost/benefit analysis and as you can see, it’s a long path forward.

So, that’s part of what’s happening with Office 365 and outlook.com right now. It’s not everything that’s going on with the service but it is a big part of it.