38 Minutes of 'This Is Not a Drill'
00:50 · Point of failure
On 13 January 2018, Hawaii sent a false ballistic missile alert during a drill. Confusing procedures and inadequate safeguards let it through; the corrective wireless alert took 38 minutes.
Editions
- Watch
- The episode on YouTube1:40
- Read
- The technical debrief on ZOF.aizof.ai
Incident
- Topics
- emergency alerts · human factors · recovery · interface design
- Point of failure
- No cancel template existed, so once the wrong alert had gone out there was no prepared way to say false alarm, and the corrective alert took 38 minutes.
Transcript
The warning
For 38 minutes, everyone in Hawaii believed a nuclear missile was incoming.
This is The Point of Failure, episode nine.
January 13, 2018, 8:07 in the morning. Every phone in Hawaii screamed at once. Ballistic missile threat inbound. Seek immediate shelter. This is not a drill.
The drill
It was a drill. During a shift change, an employee at the emergency agency was meant to run a routine test. On the screen, the test alert and the real alert sat in the same menu. Two options, side by side, one click apart.
The missing retraction
The wrong one was selected. And investigators later found the drill itself was confusingly scripted. The employee reportedly believed it was real.
Point of failure
But here is the failure that turned a mistake into 38 minutes of terror. There was no prepared way to say false alarm. No cancel template existed. They had built the alarm and never built the retraction.
So while officials scrambled, parents lowered children into storm drains. People made goodbye calls. For 38 minutes.
The safeguards
The fix afterwards took days, not years. Separate the drill from the live system. Require a second confirmation. Prepare a false alarm message in advance. Every one of those could have existed on January 12.
Every failure has a story. Every story was preventable. I'm Kevin. See you on the next one.
Sources
Report and Recommendations: Hawaii Emergency Management Agency January 13, 2018 False Alert
The investigation records the confusing drill, a generic confirmation prompt, no second-person authorization, and the 38-minute delay before the corrective wireless alert.
Related
- 07
The Typo That Broke the Internet
On 28 February 2017 an Amazon engineer ran an established playbook command to take a few S3 servers offline, one input was entered wrong, and the command removed enough capacity to take down a large part of the web for about four hours.
- 06
The Company That Locked Itself Out of Its Own Building
On 4 October 2021 a maintenance command disconnected Facebook's backbone, its DNS servers responded by withdrawing the routes that tell the internet where Facebook is, and the same outage took down the internal tools and building access the engineers needed to put it back.
- 05
The Video Game Plague That Scientists Still Study
A World of Warcraft boss attack escaped its dungeon through pets that preserved its state, turned capital cities into a virtual epidemic, and gave epidemiologists an accidental experiment in how people behave when the rules break.