The global Microsoft 365 outage was caused by a bug that had been lying dormant in a Microsoft network gateway since July. Extra load from Teams and fewer available servers brought the fault to the surface.
Last week Microsoft was hit by a global outage that left Outlook, Teams and Copilot unavailable or only partly available for almost 24 hours on 12 and 13 November. The outage began on Thursday 12 November at 17.17 CET with Exchange Online. As a result, Outlook, Teams, Copilot Chat, Intune and Purview did not work or worked poorly for hours, and recovery was very difficult.
Microsoft shares a first analysis of the outage via the Microsoft 365 admin dashboard. For those who have no access to it but still want to know what happened, Charles Lamanna, executive vice president Copilot, Agents & Platform at Microsoft, gives a condensed version via X, with sincere apologies on behalf of his employer.
Bug lay dormant since July
According to Lamanna, three factors coincided. The first is a latent fault in the servers of the network gateway, the layer through which traffic enters many Microsoft 365 services. Under certain load, lock contention arose in an internal process, where threads keep waiting for the same lock. The servers became overloaded and stopped responding.
The fault arrived with a change that Microsoft had made specifically for those gateway servers. According to Lamanna, the fault and the change do not apply to other Windows environments. Microsoft rolled out the change in phases from July, following its safe deployment procedure. After a few weeks it was running worldwide, for more than three months without problems.
The standard load tests did not catch the fault, because the test lab uses a different mix of traffic and hardware. The problem gets worse on newer servers with more cores. The tests ran mostly on weaker hardware with fewer cores.
Teams traffic and missing servers
Microsoft 365 has two independent network gateways with a different design, so that services can switch from one to the other when problems occur. About fifteen hours before the outage, Teams switched from the alternative gateway to the primary one, which also serves Exchange Online. That increased the load and changed the traffic pattern. According to Lamanna, such a switch is part of normal operations.
At the same time, almost fourteen percent of the gateway’s server capacity was out of service because of a separate problem the day before. In itself, that had no consequences, according to Lamanna, because the service had enough spare capacity.
The higher load and the reduced capacity together made the July bug suddenly resurface, during the busiest hours in the United States. Requests piled up until the servers stopped responding.
Chain reaction across the entire server fleet
When a server drops out, Microsoft’s traffic management automatically redirects users to healthy endpoints. As a result, the remaining servers received more traffic and in turn ran into the same fault. In less than two hours, the fault appeared across the entire server fleet. At the low point, more than 80 percent of the servers stopped responding.
The gateway consists of thousands of capacity units, pieces of capacity with their own hardware and network infrastructure, organised by region. Separations between those partitions are supposed to stop a chain reaction. Because of a separate problem, they did not do so this time. The analysis does not say what that problem was.
Teams, OneDrive, SharePoint and the Office apps could switch to the other gateway and therefore came back sooner. Chats and meetings in Teams worked for most users most of the time. Exchange Online cannot switch to the other gateway.
Recovery went slowly
The recovery was anything but smooth. Servers that came back online were immediately flooded again by waiting traffic and failed once more. Microsoft’s automated recovery processes are not built for a situation in which so many servers stop responding at once. Because the bug had been in production for months, a rollback did not help either.
Once it was established that the bug was the cause, the engineers developed an update that reverted the code path to the last version that worked properly. They took groups of servers out of service each time, installed the update and restarted them before putting them back into use. That happened in large blocks, sometimes more than half of a region at once.
Microsoft promises an incident report by the end of the week. The incident will make many companies think about their dependence on the company. Employees whose working day takes place largely in Teams and Outlook were virtually out of work. For one day that may produce some funny memes, but if such outages last longer or occur more often, this can seriously harm companies’ productivity.
