The Power Cut in Ghana That Saved Global Shipping
01:37 · Point of failure
On 27 June 2017 the NotPetya malware destroyed Maersk's global network within minutes, including every domain controller. As Wired reported, one copy survived in the company's Ghana office, offline after a power cut, and it seeded a rebuild the company put at about ten days.
Editions
- Watch
- The episode on YouTube2:08
- Read
- The technical debrief on ZOF.aizof.ai
Incident
- Topics
- security · recovery · resilience · dependencies
- Point of failure
- Every domain controller was synced and connected, so a backup connected to everything died with everything; the one copy that survived was offline only because of a power cut.
Transcript
The detonation
The company that moves a fifth of world shipping lost its entire network in minutes. It was saved by a power cut... in Ghana. This is The Point of Failure, episode fourteen.
June 27th, 2017. Malware called NotPetya detonates through the world's corporate networks, spreading through a poisoned software update. It does not ransom your files. It destroys them. And one of the biggest victims is Maersk: the shipping giant behind a huge share of everything you own.
Screens go black
In minutes, Maersk's screens go black around the planet. Roughly forty nine thousand laptops. Thousands of servers. Gone. Port terminals stall on multiple continents. Trucks line up outside gates that no longer know who they are.
The domain controllers
Here is the detail that decides everything. A network like Maersk's is defined by special servers called domain controllers: the master list of every user, every permission, everything. Maersk had around a hundred and fifty of them, synced worldwide. NotPetya erased every single one. Which should have been the end. Except.
The one in Accra
As reported in Wired: one domain controller survived. In the Accra office. In Ghana. Because on the day of the attack, Accra had a power outage... and that server was offline. The blackout everyone curses became the air gap that saved a global company.
The copy the disaster cannot reach
Point of failure
That single copy was hand carried across borders and became the seed to rebuild the entire network in about ten days. Total damage: around three hundred million dollars. Lesson carved in stone since: a backup connected to everything dies with everything. True resilience is the copy the disaster cannot reach.
Every failure has a story. Every story was preventable. I'm Kevin. See you at the next one.
Sources
The Untold Story of NotPetya, the Most Devastating Cyberattack in History
Wired places the surviving domain controller in a Maersk office in Ghana, without naming a city, and dates the blackout to some time before the attack. Maersk's own chief information security officer, its chief technology and information officer and its former head of identity and access management have each placed that copy in Lagos, Nigeria; in the last of those accounts a Ghana-based manager carried the disk on to the UK.
States the financial impact of the attack as USD 250-300m, most of it lost business in July and August. The episode rounds to the top of that range.
#GartnerSEC: Maersk’s Adam Banks Reflects on NotPetya Response and Recovery
The 49,000 laptops figure is the chief technology and information officer's; the chairman's figure at Davos in January 2018 was 45,000 PCs reinstalled.
Related
- 06
The Company That Locked Itself Out of Its Own Building
On 4 October 2021 a maintenance command disconnected Facebook's backbone, its DNS servers responded by withdrawing the routes that tell the internet where Facebook is, and the same outage took down the internal tools and building access the engineers needed to put it back.
- 11
The 14 Year Old Who Found Apple's Bug (And Couldn't Get Apple to Listen)
A teenager discovered that Group FaceTime could transmit audio before a call was accepted. His family struggled to report it before it became public. Apple disabled the feature and fixed the call-state logic.
- 09
38 Minutes of 'This Is Not a Drill'
On 13 January 2018, Hawaii sent a false ballistic missile alert during a drill. Confusing procedures and inadequate safeguards let it through; the corrective wireless alert took 38 minutes.