The Typo That Broke the Internet
01:02 · Point of failure
On 28 February 2017 an Amazon engineer ran an established playbook command to take a few S3 servers offline, one input was entered wrong, and the command removed enough capacity to take down a large part of the web for about four hours.
Editions
- Watch
- The episode on YouTube1:38
- Read
- The Typo That Broke a Quarter of the Webtechnical debrief · zof.ai
Incident
- System
- The Amazon S3 capacity removal playbook in the US-EAST-1 region, and the index and placement subsystems it could reach
- Date
- 28 February 2017
- Location
- Northern Virginia, United States (US-EAST-1)
- Toll
- S3 in US-EAST-1 was severely impaired for about four hours, and the sites and applications that store their files there were impaired with it. No physical harm.
- Topics
- cloud storage · capacity · blast radius · blameless postmortem · human factors
Point of failure
A routine command could take capacity from subsystems it was not aimed at, with no floor beneath which the tool would refuse, so a single mistyped input reached infrastructure the playbook was never meant to touch.
Transcript
One typo
One typo. One engineer. And a massive piece of the internet goes dark for four hours.
This is The Point of Failure, episode seven.
The storage behind everything
2017. Amazon Web Services runs S3, the storage system behind a staggering share of the internet.
Images, files, entire websites. If S3 sneezes, the internet catches a cold.
The wrong input
An engineer is debugging the billing system, following the official playbook.
Run a standard command to take a few servers offline, but one input is typed wrong.
And instead of a few servers, it pulls a huge chunk of the system out from under everything.
Websites break everywhere. Apps freeze. Smart devices go quiet.
And in the most 2017 detail imaginable: Amazon's own status dashboard cannot show the red warning icon, because the icon is stored on S3.
What actually broke
Point of failure
Here is what matters. Amazon did not fire anyone. The postmortem said the quiet part out loud: if one typo can do this, the typo is not the problem. The system is.
They rebuilt the tools so that removing too much capacity too fast is simply not possible anymore.
Humans will always make typos. The question is whether your system is designed to survive them.
Every failure has a story. Every story was preventable.
I'm Kevin. See you at the next one.
Sources
Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region
AWS's own account, and the record for almost every line of this episode. It states the command: "At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended." It states the recovery cost: "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years", with S3 operating normally again at 1:54PM PST. It states the dashboard dependency, and the change made afterwards, that the tool now removes capacity more slowly and will not take a subsystem below its minimum required capacity. It names no one.
AWS Service Health Dashboard - Feb 28, 2017 PST
The dashboard itself, captured during the outage, which is why the detail in the episode is checkable rather than a story about it. The banner reads "Increased Error Rates" and "We're continuing to work to remediate the availability issues for Amazon S3 in US-EAST-1", while the service table immediately below still reports "Service is operating normally". AWS's summary explains why: until 11:37AM PST it could not update the individual service icons "because of a dependency the SHD administration console has on Amazon S3".
Amazon AWS S3 outage is breaking things for a lot of websites and apps
Contemporaneous reporting from the day, and the record for the breadth the episode describes, including the smart devices: "Affected websites and services include Quora, newsletter provider Sailthru, Business Insider, Giphy, image hosting at a number of publisher websites, filesharing in Slack, and many more. Connected lightbulbs, thermostats and other IoT hardware is also being impacted."
AWS Outage that Broke the Internet Caused by Mistyped Command
Independent reporting on the postmortem, and on why the outage lasted as long as it did rather than ending when the mistake was found: the two affected subsystems "had not been completely restarted for many years", so the restart and its metadata safety checks "took longer than expected".
AWS says a typo caused the massive S3 failure this week
Published 2 March 2017, and the independent record of the change the episode ends on: "The tool that was responsible for causing the outage has been modified to take down servers more slowly and to block operations that will take capacity below safety check levels."
Related
- 01
The Update That Stopped the World
On 19 July 2024, a flawed CrowdStrike content update crashed Windows machines around the world. The failure was not an attack: it was a release path that trusted a validator which had accepted the configuration it was meant to stop.
- 05
The Video Game Plague That Scientists Still Study
A World of Warcraft boss attack escaped its dungeon through pets that preserved its state, turned capital cities into a virtual epidemic, and gave epidemiologists an accidental experiment in how people behave when the rules break.