The Company That Locked Itself Out of Its Own Building
01:04 · Point of failure
On 4 October 2021 a maintenance command disconnected Facebook's backbone, its DNS servers responded by withdrawing the routes that tell the internet where Facebook is, and the same outage took down the internal tools and building access the engineers needed to put it back.
Editions
- Watch
- The episode on YouTube1:44
- Read
- The Company That Deleted Itself From the Internet's Maptechnical debrief · zof.ai
Incident
- System
- Facebook's global backbone maintenance path, the DNS servers that withdrew their own route announcements, and the internal tooling that depended on both
- Date
- 4 October 2021
- Location
- Worldwide
- Toll
- Facebook, Instagram, WhatsApp and Messenger were unreachable worldwide for about six hours. No physical harm.
- Topics
- infrastructure · dns · bgp · recovery · dependencies
Point of failure
The internal tools, the communication channels and the site access the recovery depended on ran on the infrastructure that had just withdrawn itself, so the failure removed the means of repairing it.
Transcript
The disappearance
Facebook vanished from the internet for six hours. And the engineers who could fix it reportedly could not badge into their own buildings.
This is The Point of Failure, episode six.
The address book
To understand this one, you need one concept.
The internet finds websites using a kind of address book called DNS, and networks announce where they live using a protocol called BGP.
Think of it as a company shouting: here is my address, send visitors this way.
The maintenance command
October 4th, 2021. Routine maintenance on Facebook's backbone.
An engineer runs a command to check capacity. The command accidentally disconnects the data centers.
There was a safety tool designed to catch exactly this. It had a bug. It let the command through.
Facebook's systems, sensing something wrong, stopped announcing their address to the internet.
The pin gets pulled off the map. Facebook, Instagram, WhatsApp: unreachable.
For billions of people. Not crashed. Just gone. No address.
What actually broke
Point of failure
Now the twist. Facebook's internal tools ran on the same network. The systems engineers needed to fix the problem were behind the problem.
Even communications channels were down, and reporting at the time described the engineers struggling to badge into facilities because the access systems were affected too.
Six hours to walk it back.
The lesson engineers still repeat: never build your recovery tools on the same thing they are meant to recover.
Every failure has a story. Every story was preventable.
I'm Kevin. See you in the next one.
Sources
More details about the October 4 outage
Facebook's own engineering account, published 5 October 2021. It states the mechanism the episode describes: "a command was issued with the intention to assess the availability of global backbone capacity, which unintentionally took down all the connections in our backbone network, effectively disconnecting Facebook data centers globally", and "a bug in that audit tool prevented it from properly stopping the command". It also records that the DNS servers withdrew their BGP advertisements, that internal tools were lost with the network, and that the data centers are "hard to get into" so "it took extra time to activate the secure access protocols needed to get people onsite".
Understanding how Facebook disappeared from the Internet
Written the same day from Cloudflare's own BGP feed and its 1.1.1.1 resolver, so it is an outside measurement rather than a retelling. It timestamps the route withdrawals and records that "Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable." This is the record for the DNS and BGP mechanism the episode explains.
Form 10-Q for the quarterly period ended September 30, 2021
The company's own filed statement of the duration, in the risk factors of Part II, Item 1A: "For example, in October 2021, a combination of an error and a bug resulted in an approximately six-hour outage of our services." The engineering postmortem does not give a figure in hours, so this is the primary record for the six hours. Search the filing for "six-hour" to reach the sentence.
Facebook, WhatsApp, Instagram, Messenger down globally in extended service outage
Independent network measurement of the outage window: "a global outage of Facebook Inc. services including Facebook, WhatsApp, Instagram, and Messenger platforms from approximately 15:45 UTC 4 October 2021. The incident has been ongoing for six hours at the latest update." It corroborates both the scope and the length from outside the company.
Facebook, WhatsApp, Instagram suffer worldwide outage
The contemporaneous reporting the episode's badge line is careful to attribute rather than assert: "there were reports that employees at work on the company's Menlo Park, California, campus had trouble entering buildings because the outage had rendered their security badges useless." Facebook's own postmortem describes slow physical access at the data centers and says nothing about badges, so the hedge in the episode is the record, not a softening of it.