Forensic breakdowns of engineered failure
Every failure has a story.Every story was preventable.
A forensic video series on how complex systems break. Each episode traces one catastrophe back through the chain of decisions that caused it, to the exact point where the chain gave way.
Latest
Sell 1 Share... or 610,000
On 8 December 2005 a Mizuho Securities order to sell one J-Com share at 610,000 yen went out inverted, as 610,000 shares at 1 yen. The desk tried to cancel within seconds, and a defect in the Tokyo Stock Exchange's own system would not let the cancellation through.
01:18 · Point of failure
Earlier
All 8 episodes →- 07
The Typo That Broke the Internet
On 28 February 2017 an Amazon engineer ran an established playbook command to take a few S3 servers offline, one input was entered wrong, and the command removed enough capacity to take down a large part of the web for about four hours.
- 06
The Company That Locked Itself Out of Its Own Building
On 4 October 2021 a maintenance command disconnected Facebook's backbone, its DNS servers responded by withdrawing the routes that tell the internet where Facebook is, and the same outage took down the internal tools and building access the engineers needed to put it back.
- 05
The Video Game Plague That Scientists Still Study
A World of Warcraft boss attack escaped its dungeon through pets that preserved its state, turned capital cities into a virtual epidemic, and gave epidemiologists an accidental experiment in how people behave when the rules break.
- 04
The Bank That Sent $900 Million by Accident
Citibank meant to send about $7.8 million of interest on Revlon's loan. Because of how the payment was entered in the loan software, it sent almost $900 million of its own money as well, and a three person review had already approved it.
- 03
The Song That Broke YouTube's Math
Gangnam Style's view count approached the maximum value of a signed 32-bit integer. YouTube widened the counter before it overflowed, a friendly example of a failure mode that is much less friendly in systems that cannot be patched in time.
The series
Catastrophes are rarely caused by the thing that broke. They are caused by a decision made years earlier, by someone reasonable, inside a process that had no way of noticing. Each episode finds that decision, with the reports, the transcript and the primary sources on the page, so the work can be checked.
About the series →