The Point of Failure
Every failure has a story. Every story was preventable.
The Point of Failure is a forensic video series on how complex systems break. Each episode traces one catastrophe back through the chain of decisions that caused it, to the exact point where the chain gave way.
Latest
Entirely Preventable
On 7 March 2017 a critical Apache Struts vulnerability was disclosed with a patch the same day. Equifax circulated the alert but never patched the one internet facing system that needed it, neither scan run to find that system detected it, and the appliance that should have inspected the traffic had been blind for nineteen months behind an expired certificate. Attackers were inside for 76 days.
00:13 · Point of failure
Earlier
All 20 episodes →- 19
The Morning the Internet's Address Book Failed
On 19 and 20 October 2025 a latent race condition between two components of AWS's DNS automation left the DynamoDB endpoint in us-east-1 with an empty DNS record. Everything that needed that database could no longer resolve it, AWS's own systems included, and the automation was blocked from applying any further update until operators intervened by hand.
- 18
The Bank That Lost Its Customers' Money for a Month
In June 2012 an upgrade to the batch scheduler at RBS was backed out along a path nobody had tested, and the two versions turned out to be incompatible. Jobs stopped reaching the overnight queues, the backlog compounded night after night, and at least 6.5 million customers of RBS, NatWest and Ulster Bank lost reliable access to their money, some of them for weeks.
- 17
Sixteen Hours of Tay
On 23 March 2016 Microsoft released Tay, a chatbot designed to learn from the people who talked to it, onto Twitter. Within hours a coordinated group exploited that design and a repeat after me function to steer it into racist and offensive posts, and Microsoft took it offline about sixteen hours after release.
- 16
The Rocket That Lasted 37 Seconds
On 4 June 1996 the first Ariane 5 was destroyed less than a minute into its maiden flight. An alignment routine inherited from Ariane 4, with no purpose once the rocket left the pad, converted a horizontal velocity value too large for a 16 bit integer, and the backup unit ran the same software and had already failed the same way.
- 15
The AI That Deleted Production (And Then Lied About It)
In July 2025 an AI coding agent deleted a live production database during a code freeze, then told the user the data could not be restored. A rollback worked. Replit called the deletion unacceptable, refunded the user and separated development from production databases by default.
The series
Catastrophes are rarely caused by the thing that broke. They are caused by a decision made years earlier, by someone reasonable, inside a process that had no way of noticing. Each episode finds that decision, with the reports, the transcript and the primary sources on the page, so the work can be checked.
About the series →