AWS Frankfurt experiences major incident due to ‘environmental conditions’ in its Data Centre
Published 11 June 2021
Users of Amazon Web Services in Frankfurt experienced connectivity issues today (June 11). A single Availability Zone in Amazon Web Services’ EU-CENTRAL-1 Region has experienced a major incident, which in all likelihood was a fire. AWS in a Tweet said: We are investigating connectivity issues for some EC2 instances in the single Availability Zone in the EU-CENTRAL-1 Region.
Around half an hour later, AWS updated the Tweet and said: “We can confirm increased API error rates and latencies for the EC2 APIs and connectivity issues for instances within the single Availability Zone in the EU-CENTRAL-1 Region. This was caused by an increase in ambient temperature within a subsection of the affected Availability Zone. Other Availability Zones within the EU-CENTRAL-1 Region are not affected by the issue and we continue to work towards resolving the issue.”
AWS also reported that staff were unable to enter the site for safety reasons.
Later, AWS provided an update and said: “Environmental conditions within the affected Availability Zone have now returned to normal level. The vast majority of affected EC2 instances have now fully recovered but we’re continuing to work through some EBS volumes that continue to experience degraded performance.”
Kinesis Data Streams, Kinesis Firehose, Amazon Relational Database Service, and AWS CloudFormation were also impacted by this outage.
Amazon also stated that while its operators would normally had been able to restore cooling before impact, a fire suppression system activated inside a section of the affected Availability Zone.
“When this system activates, the data centre is evacuated and sealed, and a chemical is dispersed to remove oxygen from the air to extinguish any fire.” It also let clients know that the fire suppression system that activated remains disabled.
In the meantime, AWS staff had to wait for the local fire department to arrive and update on the building’s condition. “The building needed to be re-oxygenated before it was safe for engineers to enter the facility and restore the affected networking gear and servers,” said AWS.
However, there were some damages. AWS said that a very small number of remaining instances and volumes that were adversely affected by the increased ambient temperatures and loss of power remain unresolved.
AWS is not the first to experience ‘environmental conditions’. In March, one of OVHcloud’s facilities in Strasbourg (France) was ravaged by a fire.