The archive
Episodes
- 08
Sell 1 Share... or 610,000
On 8 December 2005 a Mizuho Securities order to sell one J-Com share at 610,000 yen went out inverted, as 610,000 shares at 1 yen. The desk tried to cancel within seconds, and a defect in the Tokyo Stock Exchange's own system would not let the cancellation through.
- 07
The Typo That Broke the Internet
On 28 February 2017 an Amazon engineer ran an established playbook command to take a few S3 servers offline, one input was entered wrong, and the command removed enough capacity to take down a large part of the web for about four hours.
- 06
The Company That Locked Itself Out of Its Own Building
On 4 October 2021 a maintenance command disconnected Facebook's backbone, its DNS servers responded by withdrawing the routes that tell the internet where Facebook is, and the same outage took down the internal tools and building access the engineers needed to put it back.
- 05
The Video Game Plague That Scientists Still Study
A World of Warcraft boss attack escaped its dungeon through pets that preserved its state, turned capital cities into a virtual epidemic, and gave epidemiologists an accidental experiment in how people behave when the rules break.
- 04
The Bank That Sent $900 Million by Accident
Citibank meant to send about $7.8 million of interest on Revlon's loan. Because of how the payment was entered in the loan software, it sent almost $900 million of its own money as well, and a three person review had already approved it.
- 03
The Song That Broke YouTube's Math
Gangnam Style's view count approached the maximum value of a signed 32-bit integer. YouTube widened the counter before it overflowed, a friendly example of a failure mode that is much less friendly in systems that cannot be patched in time.
- 02
$440 Million in 45 Minutes
A rollout reached seven of Knight Capital's eight production servers. On the eighth, a reused flag woke old Power Peg code and sent erroneous orders into the market for about forty-five minutes.
- 01
The Update That Stopped the World
On 19 July 2024, a flawed CrowdStrike content update crashed Windows machines around the world. The failure was not an attack: it was a release path that trusted a validator which had accepted the configuration it was meant to stop.