Entirely Preventable
00:13 · Point of failure
On 7 March 2017 a critical Apache Struts vulnerability was disclosed with a patch the same day. Equifax circulated the alert but never patched the one internet facing system that needed it, neither scan run to find that system detected it, and the appliance that should have inspected the traffic had been blind for nineteen months behind an expired certificate. Attackers were inside for 76 days.
Editions
- Watch
- The episode on YouTube2:39
- Read
- The technical debrief on ZOF.aizof.ai
Incident
- Topics
- patch management · asset inventory · certificate expiry · verification
- Point of failure
- The patch alert never reached an owner for the one internet facing system that needed it, and neither scan run to find that system detected it, so a patch that had existed for two months was never applied where it mattered.
Transcript
Entirely preventable
The patch existed for two months before the break in. The alarm system had been blind for over a year. Congress's verdict was two words: entirely preventable. This is The Point of Failure, episode twenty.
Two silent gates
Point of failure
2017. Equifax, a company most Americans never chose to do business with, holds the financial identity of nearly all of them. In March, a critical vulnerability in Apache Struts, a common web framework, is disclosed to the world, with a patch. Equifax gets the alert. Circulates it. And here the gates begin failing: the one vulnerable system that matters, a consumer dispute portal, is not on the list that gets patched. A follow up scan runs, and misses it too. Two gates. Two failures. Silent ones.
The third gate
May: attackers walk through the known, unpatched hole. And now the third gate: Equifax has monitoring designed to inspect traffic for exactly this, but the device doing the inspecting has an expired certificate, expired, per the investigation, some nineteen months earlier, which quietly blinds it. The attackers operate inside for seventy six days. When someone finally renews that certificate, the suspicious traffic lights up almost immediately. The alarm worked. It had been unplugged by paperwork.
What was taken
One hundred forty seven million people's data, Social Security numbers, birth dates, the unchangeable keys of financial life, gone. The CEO resigned. The settlement reached toward seven hundred million dollars. And the House investigation delivered the epitaph this whole arc is built on: entirely preventable. Not sophisticated. Not inevitable. Preventable.
Verified tools protect you
Because look at what actually failed: not cryptography, not firewalls, verification. Does the asset inventory match reality? Did the patch actually land? Does the scanner actually see? Is the monitoring actually monitoring? Every one is a test nobody ran, on the process itself. Security tools do not protect you. Verified security tools protect you. The difference was a hundred forty seven million people.
Every failure has a story. Every story was preventable. I'm Kevin. See you at the next one.
Sources
The source of the phrase, and of the nineteen months: the certificate on the appliance monitoring the dispute portal expired on 31 January 2016, and the report rejects the ten months reported elsewhere on the strength of the company's own list of expired certificates. Read it for three things the episode compresses. It is a majority staff report of one House committee rather than a finding of Congress, and the minority published separately. Its own figures are 143 million announced, later grown to 148 million. And its word for the departures is retired, in its own quotation marks, not resigned. It describes web shells and encrypted exfiltration and nowhere calls the intrusion unsophisticated; what it calls preventable is the failure to patch.
Where the figure of 147 million comes from; it is the FTC's, not the House committee's. The settlement is at least 575 million dollars and potentially up to 700 million, so the larger number is a ceiling that depends on a top up rather than an amount paid.
Data Protection: Actions Taken by Equifax and Federal Agencies in Response to the 2017 Breach
The second federal account, and the one that explains how the alert failed: the recipient list was out of date, so the notice was not received by the people who would have installed the patch. It puts the certificate at about ten months rather than nineteen, which is the figure the House report examined and rejected.
Form 10-K for the fiscal year ended December 31, 2018
The company's own account: unauthorised access from mid May 2017 through July 2017, approximately 145.5 million United States consumers affected, and the vulnerability that was exploited was not identified by its security processes. It never uses the figure 147 million.
The patch was published on 7 March 2017, the same day the vulnerability was announced. The intrusion began on 13 May, 67 days later, which is what the two months in the opening line refers to.
Related
- 12
The Expired Certificate That Silenced 11 Countries
On 6 December 2018, an expired certificate in Ericsson core-network software disrupted operators across 11 countries. O2 and SoftBank were among the networks affected by the same dated dependency.
- 18
The Bank That Lost Its Customers' Money for a Month
In June 2012 an upgrade to the batch scheduler at RBS was backed out along a path nobody had tested, and the two versions turned out to be incompatible. Jobs stopped reaching the overnight queues, the backlog compounded night after night, and at least 6.5 million customers of RBS, NatWest and Ulster Bank lost reliable access to their money, some of them for weeks.
- 17
Sixteen Hours of Tay
On 23 March 2016 Microsoft released Tay, a chatbot designed to learn from the people who talked to it, onto Twitter. Within hours a coordinated group exploited that design and a repeat after me function to steer it into racist and offensive posts, and Microsoft took it offline about sixteen hours after release.