https://thepointoffailure.com 2026-09-26T23:51:54.000Z weekly 1 https://thepointoffailure.com/episodes 2026-09-26T23:51:54.000Z weekly 0.9 https://thepointoffailure.com/about 2026-09-19 yearly 0.5 https://thepointoffailure.com/privacy 2026-08-29 yearly 0.2 https://thepointoffailure.com/terms 2026-08-29 yearly 0.2 https://thepointoffailure.com/episodes/020-entirely-preventable Episode 20: Entirely Preventable https://i.ytimg.com/vi/xzzInh47TTA/maxresdefault.jpg On 7 March 2017 a critical Apache Struts vulnerability was disclosed with a patch the same day. Equifax circulated the alert but never patched the one internet facing system that needed it, neither scan run to find that system detected it, and the appliance that should have inspected the traffic had been blind for nineteen months behind an expired certificate. Attackers were inside for 76 days. https://www.youtube-nocookie.com/embed/xzzInh47TTA 159 2026-09-26T23:51:54.000Z yes no The Point of Failure 2026-09-26T23:51:54.000Z monthly 0.8 https://thepointoffailure.com/episodes/019-the-morning-the-internet-s-address-book-failed Episode 19: The Morning the Internet's Address Book Failed https://i.ytimg.com/vi/GgEeIl0qIYw/maxresdefault.jpg On 19 and 20 October 2025 a latent race condition between two components of AWS's DNS automation left the DynamoDB endpoint in us-east-1 with an empty DNS record. Everything that needed that database could no longer resolve it, AWS's own systems included, and the automation was blocked from applying any further update until operators intervened by hand. https://www.youtube-nocookie.com/embed/GgEeIl0qIYw 139 2026-09-24T03:38:48.000Z yes no The Point of Failure 2026-09-24T03:38:48.000Z monthly 0.8 https://thepointoffailure.com/episodes/018-the-bank-that-lost-its-customers-money-for-a-month Episode 18: The Bank That Lost Its Customers' Money for a Month https://i.ytimg.com/vi/8hUie2GO1gY/maxresdefault.jpg In June 2012 an upgrade to the batch scheduler at RBS was backed out along a path nobody had tested, and the two versions turned out to be incompatible. Jobs stopped reaching the overnight queues, the backlog compounded night after night, and at least 6.5 million customers of RBS, NatWest and Ulster Bank lost reliable access to their money, some of them for weeks. https://www.youtube-nocookie.com/embed/8hUie2GO1gY 101 2026-09-24T03:38:15.000Z yes no The Point of Failure 2026-09-24T03:38:15.000Z monthly 0.8 https://thepointoffailure.com/episodes/017-sixteen-hours-of-tay Episode 17: Sixteen Hours of Tay https://i.ytimg.com/vi/RP5H4ezc8BM/maxresdefault.jpg On 23 March 2016 Microsoft released Tay, a chatbot designed to learn from the people who talked to it, onto Twitter. Within hours a coordinated group exploited that design and a repeat after me function to steer it into racist and offensive posts, and Microsoft took it offline about sixteen hours after release. https://www.youtube-nocookie.com/embed/RP5H4ezc8BM 143 2026-09-20T20:54:32.000Z yes no The Point of Failure 2026-09-20T20:54:32.000Z monthly 0.8 https://thepointoffailure.com/episodes/016-the-rocket-that-lasted-37-seconds Episode 16: The Rocket That Lasted 37 Seconds https://i.ytimg.com/vi/iy-vjB73bVo/maxresdefault.jpg On 4 June 1996 the first Ariane 5 was destroyed less than a minute into its maiden flight. An alignment routine inherited from Ariane 4, with no purpose once the rocket left the pad, converted a horizontal velocity value too large for a 16 bit integer, and the backup unit ran the same software and had already failed the same way. https://www.youtube-nocookie.com/embed/iy-vjB73bVo 125 2026-09-19T07:34:25.000Z yes no The Point of Failure 2026-09-19T07:34:25.000Z monthly 0.8 https://thepointoffailure.com/episodes/015-the-ai-that-deleted-production-and-then-lied-about-it Episode 15: The AI That Deleted Production (And Then Lied About It) https://i.ytimg.com/vi/p4OFY_LGYBA/maxresdefault.jpg In July 2025 an AI coding agent deleted a live production database during a code freeze, then told the user the data could not be restored. A rollback worked. Replit called the deletion unacceptable, refunded the user and separated development from production databases by default. https://www.youtube-nocookie.com/embed/p4OFY_LGYBA 129 2026-09-14T12:25:51.000Z yes no The Point of Failure 2026-09-14T12:25:51.000Z monthly 0.8 https://thepointoffailure.com/episodes/014-the-power-cut-in-ghana-that-saved-global-shipping Episode 14: The Power Cut in Ghana That Saved Global Shipping https://i.ytimg.com/vi/yYkoVtstQTI/maxresdefault.jpg On 27 June 2017 the NotPetya malware destroyed Maersk's global network within minutes, including every domain controller. As Wired reported, one copy survived in the company's Ghana office, offline after a power cut, and it seeded a rebuild the company put at about ten days. https://www.youtube-nocookie.com/embed/yYkoVtstQTI 128 2026-09-14T00:10:47.000Z yes no The Point of Failure 2026-09-14T00:10:47.000Z monthly 0.8 https://thepointoffailure.com/episodes/013-the-fan-who-fixed-a-billion-dollar-game Episode 13: The Fan Who Fixed a Billion Dollar Game https://i.ytimg.com/vi/F0MacjkDjVg/maxresdefault.jpg For seven years GTA Online could take minutes to load. In 2021 a programmer working without the source code traced most of the wait on his PC to one in-game item list being read inefficiently. Rockstar confirmed it, shipped a fix for the PC version and paid a bounty. https://www.youtube-nocookie.com/embed/F0MacjkDjVg 122 2026-09-13T15:53:17.000Z yes no The Point of Failure 2026-09-13T15:53:17.000Z monthly 0.8 https://thepointoffailure.com/episodes/012-the-expired-certificate-that-silenced-11-countries Episode 12: The Expired Certificate That Silenced 11 Countries https://i.ytimg.com/vi/Z2hj49kU56Y/maxresdefault.jpg On 6 December 2018, an expired certificate in Ericsson core-network software disrupted operators across 11 countries. O2 and SoftBank were among the networks affected by the same dated dependency. https://www.youtube-nocookie.com/embed/Z2hj49kU56Y 100 2026-09-10T02:18:00.000Z yes no The Point of Failure 2026-09-10T02:18:00.000Z monthly 0.8 https://thepointoffailure.com/episodes/011-the-14-year-old-who-found-apple-s-bug-and-couldn-t-get-apple Episode 11: The 14 Year Old Who Found Apple's Bug (And Couldn't Get Apple to Listen) https://i.ytimg.com/vi/mjrPmfw3XYQ/maxresdefault.jpg A teenager discovered that Group FaceTime could transmit audio before a call was accepted. His family struggled to report it before it became public. Apple disabled the feature and fixed the call-state logic. https://www.youtube-nocookie.com/embed/mjrPmfw3XYQ 109 2026-09-09T10:50:32.000Z yes no The Point of Failure 2026-09-09T10:50:32.000Z monthly 0.8 https://thepointoffailure.com/episodes/010-the-disaster-that-never-happened Episode 10: The Disaster That Never Happened https://i.ytimg.com/vi/80onoCAOZJE/maxresdefault.jpg Two-digit years made the millennium rollover a real software risk. Years of remediation and testing helped keep the disruption limited, leaving successful prevention looking like an unnecessary warning. https://www.youtube-nocookie.com/embed/80onoCAOZJE 107 2026-09-08T20:13:01.000Z yes no The Point of Failure 2026-09-08T20:13:01.000Z monthly 0.8 https://thepointoffailure.com/episodes/009-38-minutes-of-this-is-not-a-drill Episode 09: 38 Minutes of 'This Is Not a Drill' https://i.ytimg.com/vi/fLssMse1IbA/maxresdefault.jpg On 13 January 2018, Hawaii sent a false ballistic missile alert during a drill. Confusing procedures and inadequate safeguards let it through; the corrective wireless alert took 38 minutes. https://www.youtube-nocookie.com/embed/fLssMse1IbA 100 2026-09-07T20:01:00.000Z yes no The Point of Failure 2026-09-07T20:01:00.000Z monthly 0.8 https://thepointoffailure.com/episodes/008-sell-1-share-or-610-000 Episode 08: Sell 1 Share... or 610,000 https://i.ytimg.com/vi/KZC2IoQQQY0/maxresdefault.jpg The order was for far more shares than J-Com had in existence. A warning was shown and overridden, the exchange accepted the order, and repeated cancellation attempts failed because of a flaw in the exchange's trading system. Mizuho's loss ran to about 40.7 billion yen, and Japanese courts later held the exchange partly liable for the system defect. https://www.youtube-nocookie.com/embed/KZC2IoQQQY0 102 2026-09-06T16:22:59.000Z yes no The Point of Failure 2026-09-06T16:22:59.000Z monthly 0.8 https://thepointoffailure.com/episodes/007-the-typo-that-broke-the-internet Episode 07: The Typo That Broke the Internet https://i.ytimg.com/vi/QMjusfXFTtI/maxresdefault.jpg A debugging command aimed at the S3 billing subsystem was entered with an incorrect input and removed a much larger set of servers than intended, including servers behind two critical subsystems in US-EAST-1. The index subsystem had not been fully restarted in years and the restart took hours. AWS published the account and rebuilt the tool so the same removal could not happen again. https://www.youtube-nocookie.com/embed/QMjusfXFTtI 98 2026-09-04T18:47:26.000Z yes no The Point of Failure 2026-09-04T18:47:26.000Z monthly 0.8 https://thepointoffailure.com/episodes/006-the-company-that-locked-itself-out-of-its-own-building Episode 06: The Company That Locked Itself Out of Its Own Building https://i.ytimg.com/vi/wFWfVxX6i3E/maxresdefault.jpg A command meant to audit backbone capacity disconnected Facebook's data centers from each other, and an audit tool built to block exactly that command failed to. The DNS servers that could no longer reach those data centers withdrew their own BGP routes, so Facebook, Instagram and WhatsApp had no address for about six hours. The recovery ran on the same infrastructure. https://www.youtube-nocookie.com/embed/wFWfVxX6i3E 104 2026-09-03T10:38:16.000Z yes no The Point of Failure 2026-09-03T10:38:16.000Z monthly 0.8 https://thepointoffailure.com/episodes/005-the-video-game-plague-that-scientists-still-study Episode 05: The Video Game Plague That Scientists Still Study https://i.ytimg.com/vi/IXtm7CZ9moE/maxresdefault.jpg In 2005, Corrupted Blood was meant to remain inside World of Warcraft's Zul'Gurub raid. Pets carried it out after being dismissed and resummoned, so it spread through cities, defeated a voluntary quarantine, and exposed the behavior an epidemic model needs to account for. https://www.youtube-nocookie.com/embed/IXtm7CZ9moE 95 2026-09-01T18:27:42.000Z yes no The Point of Failure 2026-09-01T18:27:42.000Z monthly 0.8 https://thepointoffailure.com/episodes/004-the-bank-that-sent-900-million-by-accident Episode 04: The Bank That Sent $900 Million by Accident https://i.ytimg.com/vi/EmXJQmZ5d6k/maxresdefault.jpg On 11 August 2020 Citibank, the administrative agent on a syndicated term loan to Revlon, intended to wire approximately $7.8 million of interest and sent the principal as well, a payment its own notice recorded as 893,944,008.52. A maker, a checker and an approver had each passed the transaction. Lenders holding roughly $500 million refused to return it, and a court allowed them to keep it before that judgment was vacated on appeal. https://www.youtube-nocookie.com/embed/EmXJQmZ5d6k 116 2026-08-30T18:35:36.000Z yes no The Point of Failure 2026-08-30T18:35:36.000Z monthly 0.8 https://thepointoffailure.com/episodes/003-the-song-that-broke-youtube-s-math Episode 03: The Song That Broke YouTube's Math https://i.ytimg.com/vi/nrBuDELYsV8/maxresdefault.jpg The number 2,147,483,647 is the largest positive value a signed 32-bit integer can represent. As Gangnam Style approached it in 2014, YouTube upgraded its counter to 64 bits. The counter did not break; the story is about an assumption that had to be retired before reality reached it. https://www.youtube-nocookie.com/embed/nrBuDELYsV8 93 2026-08-27T12:20:30.000Z yes no The Point of Failure 2026-08-27T12:20:30.000Z monthly 0.8 https://thepointoffailure.com/episodes/002-440-million-in-45-minutes Episode 02: $440 Million in 45 Minutes https://i.ytimg.com/vi/IwjPzC8HNYU/maxresdefault.jpg Knight Capital's 2012 market-access failure was a chain: old code remained on a production server, a flag was repurposed, one server missed the deployment, and no verification proved the fleet matched. The SEC later found inadequate controls around erroneous orders, deployment, and incident response. https://www.youtube-nocookie.com/embed/IwjPzC8HNYU 112 2026-08-27T00:26:48.000Z yes no The Point of Failure 2026-08-27T00:26:48.000Z monthly 0.8 https://thepointoffailure.com/episodes/001-the-update-that-stopped-the-world Episode 01: The Update That Stopped the World https://i.ytimg.com/vi/Hmt4yV3huK0/maxresdefault.jpg CrowdStrike's Falcon sensor received a Rapid Response Content update containing problematic data. The resulting exception crashed affected Windows systems, and recovery often required hands-on work at each machine. This episode traces the gap between a validation check passing and a configuration being safe to deploy. https://www.youtube-nocookie.com/embed/Hmt4yV3huK0 87 2026-08-26T22:17:01.000Z yes no The Point of Failure 2026-08-26T22:17:01.000Z monthly 0.8