Updated

Microsoft Azure Incident

What happened?
Between as early as 08:00 UTC and 13:20 UTC on 29 July 2022, customers may have experienced connectivity issues such as network drops, latency, and/or degradation when attempting to access or manage Azure resources in multiple regions. Most of the impact was determined to have occurred in the following regions – Japan East, Canada Central, Brazil South, East Asia, East US, South Central US, East US 2, West US, West Europe, France Central, North Central US, South Africa North, Korea Central, and Southeast Asia.

What went wrong, and why?
Starting at 08:00 UTC on 29 July, the Azure WAN network began to experience a significant spike of traffic, upwards of 60 Tbps. While the event was detected immediately and automated remediation was triggered, the size of the spike impacted the ability of automatic mitigations to handle the volume. This impacted both intra-region and cross-region traffic over various network paths which included ExpressRoute.

How did Microsoft respond?
We have developed several detection and mitigation algorithms that were triggered and alerted upon automatically around 08:00 UTC when the traffic spike started. However, the level of the spike was an unprecedented level of traffic not seen previously. So, while the mitigation mechanisms were successfully triggered, the substantial volume of traffic caused the mitigations to take longer to return traffic levels to normal. The traffic returned to normal at 13:30 UTC without additional remediations applied, which is when impact to customers would have been mitigated for this event.