Updated Friday morning. Outlook is loading again for us, and Microsoft says service is improving in Europe and North America. As of this update it still hasn’t called the incident resolved. We’ll update again when it does, and again when the root cause comes out.
Outlook went down shortly after noon Eastern on Thursday. Mailboxes stopped loading and mail stopped sending for a lot of people across the US.
Microsoft is tracking it as incident EX1550543 in the Microsoft 365 admin center. Its first word on X was that it was “investigating reports of users experiencing issues accessing Exchange Online mailboxes” in the Americas. Then it widened that: the issue “may affect Microsoft 365 service connectivity for users in any region.”
Downdetector complaints went past 65,000 at one point, according to WGCU. CNBC later put it around 130,000. Those are reports, not affected users, so treat any single figure loosely.
What Microsoft said Thursday
Microsoft first blamed “elevated load across multiple regions.” It paused all deployments and started moving traffic off the affected infrastructure.
Then it got more specific. The load wasn’t extra traffic. It came from a “number of servers being unexpectedly unavailable; straining the remaining infrastructure,” according to CNBC. Some of those servers were failing automated health checks.
Later in the afternoon Microsoft said it had “validated an alternative service health check method” that brought some of the servers back online. Servers it returned to service could still “become unresponsive again once they receive traffic,” per ABC News.
Overnight and Friday morning
The night didn’t go smoothly. At one point Microsoft said its attempts “to improve serviceability” were “not taking as expected,” according to NBC News.
Around 2:30 a.m. Eastern it reported “positive results for server stability after applying an upstream authentication/routing configuration component patch.” That’s the first time Microsoft has pointed at a specific component. It also said it’s “not aware of any malicious activity that is causing this incident.”
By about 7 a.m. Eastern it was describing “encouraging recovery progress in Europe,” and said “North America is also improving; however, some users may still experience intermittent issues.” It also extended an email timeout window to 48 hours so delayed messages don’t get dropped, per NBC.
On Thursday our read was that the health checks themselves were part of the problem, since healthy capacity kept getting pulled out of rotation. The overnight patch to an authentication and routing component suggests the trouble sat further upstream than that. Microsoft hasn’t connected those dots, so treat this as our interpretation. It says full root cause details will come in a post-incident report “once mitigated.”
We’ve seen this one before
In January, Microsoft 365 went down for almost ten hours. Outlook, Defender and Purview were all hit, and external mail stopped for a lot of tenants. Microsoft’s explanation then, per The Register, was “a portion of service infrastructure in North America that is not processing traffic as expected.”
This week’s language is close to that. Different incident, same shape: a chunk of infrastructure misbehaves and everybody downstream finds out at once. This one ran a lot longer, though. It was still going 17 hours in.
If you’re an admin this morning
- Keep an eye on EX1550543 in the admin center until Microsoft marks it resolved. Microsoft says intermittent issues can still show up.
- Expect a pile of delayed mail to arrive late. Microsoft stretched the timeout to 48 hours, so messages sent during the outage should still land instead of bouncing.
- Spot check mail flow in both directions, inbound from outside especially. After January some users kept seeing problems after Microsoft called it resolved.
- If anything time sensitive went out Thursday afternoon, confirm the other side actually got it.
- Don’t start resetting passwords or rebuilding Outlook profiles for people who still see errors. Give it some time first, since the problem has been on Microsoft’s side the whole way.
Where that leaves us
Nobody on 365 did anything wrong this week. That’s the deal when your email lives on somebody else’s servers. Their bad day is your bad day, and you find out from Downdetector.
Running your own mail server instead is a real job, and it goes down too. For most people Microsoft is still the better bet. The part worth deciding now, while it’s fresh, is how you’d reach people the next time this happens. Even a group text works, as long as it doesn’t depend on the same login.
Microsoft will publish a post-incident review for EX1550543 in the admin center. When it does, we’ll update this with the cause.
Sources: WGCU, CNBC, ABC News, NBC News and The Register.