I’d like to give you an update on the
announcement that I posted last week. Late yesterday (September 30th),
we completed a reboot of less than 10% of the EC2 fleet to protect you from any
security risks associated with the Xen Security Advisory
This Xen Security Advisory was embargoed until a few minutes ago; we
were obligated to keep all information about the issue confidential until it was
published. The Xen community (in which we are active participants)
has designed a two-stage disclosure process that operates as follows:
Early disclosure to select organizations (a list maintained
and regularly evaluated by the Xen
Security Team based on a set of public criteria established by
the Xen Project community) with a limited time to make
accommodations and apply updates before it becomes widely known.
- Full disclosure to everyone on the public disclosure date.
Because our customers’ security is our top priority and because the
issue was potentially harmful to our customers, we needed to take
fast action to protect them. For the reasons mentioned above,
we couldn’t be as expansive as we’d have liked on why we had to take such fast action.
The zone by zone reboots were completed as planned and we worked very closely
with our customers to ensure that the reboots went smoothly for them.
We’ll continue to be vigilant and will do our best to protect all
AWS customers from similar issues in the future. As an AWS user, you
may also want to take this opportunity to re-examine your AWS architecture
to look for possible ways to make it even more fault-tolerant.
Here are a few suggestions to get you started:
Run instances in two or more
Pay attention to your Inbox and to the alerts on the AWS Management Console.
Make sure that you fill in the “Alternate Contacts” in the AWS Billing Console.
Review the personalized assessment of your architecture in the
Trusted Advisor, then open up AWS Premium Support Cases to get engineering
assistance as you implement architectural best practices.
Monkey to induce various kinds of failures in a controlled
Examine and consider expanding your use of Amazon Route 53 and Elastic Load Balancing
checks to ensure that web traffic is routed to healthy instances.
- Use Auto Scaling to keep a defined number of healthy instances up and running.
You should also consult our
Overview of Security Practices
whitepaper for more information around AWS and security.