An error in Azure and Microsoft 365 is due to an outage in the maintenance system.
Microsoft has announced the cause of the major outage that affected several Azure and Microsoft 365 services last week. According to the company, a software bug in an automated maintenance system caused network connections to be removed on many more devices than intended.
The outage primarily affected customers using Microsoft 365 services via network infrastructure connected to the West US Azure region. A few hours later, the issues were resolved after Microsoft rolled back the change.
Error during routine maintenance
According to a preliminary incident report, Microsoft was performing scheduled maintenance where certain network paths were to be temporarily isolated. The maintenance system automatically converts those requests into instructions for the network equipment and normally checks whether a redundant connection remains available at all times. However, due to a software bug, additional network devices were incorrectly included in the maintenance.
As a result, IP routes disappeared on more systems than anticipated, disrupting traffic between the West US data center and the broader Microsoft network. Network traffic that remained entirely within the West US region was not affected, according to Microsoft.
The network issues caused connectivity problems and higher latency for a large number of Microsoft services. Within Microsoft 365, OneDrive, SharePoint Online, Microsoft Teams, Power Automate, Copilot Chat, Microsoft Loop, and the Microsoft 365 Admin Center were affected. Services such as Power BI, Fabric, Windows 365, Copilot Studio, and Microsoft Defender also experienced disruptions. On the Azure side, Azure Kubernetes Service (AKS), Azure Cosmos DB, Azure Databricks, Azure Firewall, Azure Monitor, Azure Virtual Desktop, Microsoft Sentinel, and ExpressRoute were among those hit.
Additional checks announced
Microsoft is now investigating how the error occurred and intends to tighten security checks around automated maintenance processes. The company will publish a final Post Incident Review within the next two weeks with a full analysis and any additional measures to prevent similar incidents in the future.
