Well I am writing this one after AWS us-east-1 aka North Virginia region took a hit from one of its developers try to deploy few days back. This made me realize the reliability of AWS infrastructure.
We host all our infrastructure in the us-east-1 for the obvious cost benefits. North Virginia region of AWS is the first & the oldest data center. This is where all the new features are tested and rolled out first. In fact it hosts a majority of AWS clients here at the time of this writing.
Last month while I was heading home, my phone which was kept in my car cockpit was continuously buzzing with alerts. It was a usual scenario on Friday as our biggest customer traffic surged with the onset of weekend. While I was waiting for the light to turn green, I noticed that there were some unusual number of alerts. But I did not gave a damn because no phone while driving. #safedriving
Reached home. Got into the bed and was about to sleep when my phone popped a notification from Facebook. Logged on and it was a comment on one of the older post. Suddenly the FB app crashed and reloaded again with the updated timeline. And there it was. A post by TechCrunch that AWS us-east-1 was on fire.
Bingo ! Logged on my workstation and started checking the alerts. AWS Auto-scaling groups were unable to spin new instances, backups failed, new instances not being launched. I checked the traffic on our load-balancer and it was all red with 5xx errors. I tried to launch instance manually using the golden ami but without success. Everything was off. Business was hampered. Called my boss and notified him. We looked into our disaster recovery scenario but there was none configured. We notified all the stakeholders. This was a business loss case.
Saturday early morning the region recovered and everything was working as usual. It was time for an Root-Cause-Analysis. Started analyzing with the AWS documentation of the issue and realized that the S3 service was hit. This was due to a manual error where an update was rolled out to the entire fleet of infrastructure instead of a small portion. This explained the issue with ASGs and the reason why I was unable to launch new instances. Amazon AMIs are stored in S3 internally and due to the update taking down the S3 service itself, AMIs were inaccessible. Backups for RDS service are also stored in S3 and hence they failed as well.
This made me realize two things:
Reliability of North Virginia (us-east-1)
This is the oldest and by far the largest region of AWS. As of this writing it serves most of the AWS customers. This is also the cheapest region to run your infrastructure on AWS. For this reason, the load is always high here. Many of the Availability Zones at times goes down intermittently in this region. AWS consultants also persuade you to move out of this zone unless your business is based out of North Virginia. Also any new feature is always rolled out and tested in this region. Hence if possible you should try not to host your resources in this region.
Multi Region Infrastructure / Cross Region Replication
Even if you are not hosted in North Virginia, chances are very high that the entire region might go off. As per the AWS well architected framework, you should always have a DR site for your infrastructure. Remember the good old days of DCs where you would set up multiple Data Centers for Disaster Recovery. Yes ! That is exactly what we need, especially if your business is a 24×7 system which cannot afford unplanned downtime. As of now Amazon might not be offering any built in services for cross-region replication but with a slight increase in costs parallel infrastructure can be run as a workaround to this issue.
I am pretty sure AWS would come up with a solution to this very soon !
Till then make sure you stay away from North Virginia, no not the place but the AWS region and if it is hard to steer clear of it ensure that you have a backup region where the infrastructure can be re-created in the eventuality of any failure.
