2017-02-28

Handling the AWS US-EAST-1 outage

The ongoing outage in AWS US-EAST-1 (N. Virginia) affected a number of Aiven users, but the combination of our automation and manual actions taken by the operations team has resolved the issues for all Aiven users.

Our 24/7 monitoring alerted us to the outage, initially thought to be limited to S3 but later revealed to affect more AWS resources, including EBS volumes and launching new EC2 instances.  A number of Aiven services running on affected EBS volumes in affected availability zones were migrated online to new instances in AWS US-EAST-2 (Ohio) ensuring service availability while the N. Virginia region is experiencing issues.

There was also a number of Aiven services running on EC2 instances that continued to operate normally, but where access to S3 for backups started timing out and failing.   The affected services have been updated to store backups to the Ohio region until the N. Virginia region recovers.

This ensures that all Aiven services have proper backups during the outage.  It won't be possible to spin up new instances in N. Virginia or perform a PostgreSQL point-in-time-recovery to a backup location stored only in N. Virginia while the outage is ongoing, but once AWS has resolved the issue Aiven services will be automatically restored.

Update: Please see our help site for more details about the outage.

Cheers,
Team Aiven

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.