The archive
Every episode of The Point of Failure
Every episode on this page carries the full transcript with timecodes, the incident record, and the primary sources it was built from. Newest first.
- 20
Entirely Preventable
On 7 March 2017 a critical Apache Struts vulnerability was disclosed with a patch the same day. Equifax circulated the alert but never patched the one internet facing system that needed it, neither scan run to find that system detected it, and the appliance that should have inspected the traffic had been blind for nineteen months behind an expired certificate. Attackers were inside for 76 days.
- 19
The Morning the Internet's Address Book Failed
On 19 and 20 October 2025 a latent race condition between two components of AWS's DNS automation left the DynamoDB endpoint in us-east-1 with an empty DNS record. Everything that needed that database could no longer resolve it, AWS's own systems included, and the automation was blocked from applying any further update until operators intervened by hand.
- 18
The Bank That Lost Its Customers' Money for a Month
In June 2012 an upgrade to the batch scheduler at RBS was backed out along a path nobody had tested, and the two versions turned out to be incompatible. Jobs stopped reaching the overnight queues, the backlog compounded night after night, and at least 6.5 million customers of RBS, NatWest and Ulster Bank lost reliable access to their money, some of them for weeks.
- 17
Sixteen Hours of Tay
On 23 March 2016 Microsoft released Tay, a chatbot designed to learn from the people who talked to it, onto Twitter. Within hours a coordinated group exploited that design and a repeat after me function to steer it into racist and offensive posts, and Microsoft took it offline about sixteen hours after release.
- 16
The Rocket That Lasted 37 Seconds
On 4 June 1996 the first Ariane 5 was destroyed less than a minute into its maiden flight. An alignment routine inherited from Ariane 4, with no purpose once the rocket left the pad, converted a horizontal velocity value too large for a 16 bit integer, and the backup unit ran the same software and had already failed the same way.
- 15
The AI That Deleted Production (And Then Lied About It)
In July 2025 an AI coding agent deleted a live production database during a code freeze, then told the user the data could not be restored. A rollback worked. Replit called the deletion unacceptable, refunded the user and separated development from production databases by default.
- 14
The Power Cut in Ghana That Saved Global Shipping
On 27 June 2017 the NotPetya malware destroyed Maersk's global network within minutes, including every domain controller. As Wired reported, one copy survived in the company's Ghana office, offline after a power cut, and it seeded a rebuild the company put at about ten days.
- 13
The Fan Who Fixed a Billion Dollar Game
For seven years GTA Online could take minutes to load. In 2021 a programmer working without the source code traced most of the wait on his PC to one in-game item list being read inefficiently. Rockstar confirmed it, shipped a fix for the PC version and paid a bounty.
- 12
The Expired Certificate That Silenced 11 Countries
On 6 December 2018, an expired certificate in Ericsson core-network software disrupted operators across 11 countries. O2 and SoftBank were among the networks affected by the same dated dependency.
- 11
The 14 Year Old Who Found Apple's Bug (And Couldn't Get Apple to Listen)
A teenager discovered that Group FaceTime could transmit audio before a call was accepted. His family struggled to report it before it became public. Apple disabled the feature and fixed the call-state logic.
- 10
The Disaster That Never Happened
Two-digit years made the millennium rollover a real software risk. Years of remediation and testing helped keep the disruption limited, leaving successful prevention looking like an unnecessary warning.
- 09
38 Minutes of 'This Is Not a Drill'
On 13 January 2018, Hawaii sent a false ballistic missile alert during a drill. Confusing procedures and inadequate safeguards let it through; the corrective wireless alert took 38 minutes.
- 08
Sell 1 Share... or 610,000
On 8 December 2005 a Mizuho Securities order to sell one J-Com share at 610,000 yen went out inverted, as 610,000 shares at 1 yen. The desk tried to cancel within seconds, and a defect in the Tokyo Stock Exchange's own system would not let the cancellation through.
- 07
The Typo That Broke the Internet
On 28 February 2017 an Amazon engineer ran an established playbook command to take a few S3 servers offline, one input was entered wrong, and the command removed enough capacity to take down a large part of the web for about four hours.
- 06
The Company That Locked Itself Out of Its Own Building
On 4 October 2021 a maintenance command disconnected Facebook's backbone, its DNS servers responded by withdrawing the routes that tell the internet where Facebook is, and the same outage took down the internal tools and building access the engineers needed to put it back.
- 05
The Video Game Plague That Scientists Still Study
A World of Warcraft boss attack escaped its dungeon through pets that preserved its state, turned capital cities into a virtual epidemic, and gave epidemiologists an accidental experiment in how people behave when the rules break.
- 04
The Bank That Sent $900 Million by Accident
Citibank meant to send about $7.8 million of interest on Revlon's loan. Because of how the payment was entered in the loan software, it sent almost $900 million of its own money as well, and a three person review had already approved it.
- 03
The Song That Broke YouTube's Math
Gangnam Style's view count approached the maximum value of a signed 32-bit integer. YouTube widened the counter before it overflowed, a friendly example of a failure mode that is much less friendly in systems that cannot be patched in time.
- 02
$440 Million in 45 Minutes
A rollout reached seven of Knight Capital's eight production servers. On the eighth, a reused flag woke old Power Peg code and sent erroneous orders into the market for about forty-five minutes.
- 01
The Update That Stopped the World
On 19 July 2024, a flawed CrowdStrike content update crashed Windows machines around the world. The failure was not an attack: it was a release path that trusted a validator which had accepted the configuration it was meant to stop.