The Bank That Lost Its Customers' Money for a Month
00:27 · Point of failure
In June 2012 an upgrade to the batch scheduler at RBS was backed out along a path nobody had tested, and the two versions turned out to be incompatible. Jobs stopped reaching the overnight queues, the backlog compounded night after night, and at least 6.5 million customers of RBS, NatWest and Ulster Bank lost reliable access to their money, some of them for weeks.
Editions
- Watch
- The episode on YouTube1:41
- Read
- The technical debrief on ZOF.aizof.ai
Incident
- Topics
- batch processing · change management · testing · banking
- Point of failure
- An upgrade to the batch scheduler was backed out along a path nobody had tested, and the two versions were incompatible, so jobs stopped reaching the overnight queues.
Transcript
Locked out
Six and a half million people. Locked out of their own money. For weeks. Because one overnight job failed. This is The Point of Failure, episode eighteen.
The nightly run
Here is a secret about banks. Your balance is not updated the moment things happen. Every night, a giant batch process runs: wages in, bills out, millions of transactions applied while you sleep. At RBS, one of Britain's biggest banking groups, that nightly run was the heartbeat.
The upgrade and the backout
Point of failure
June 2012. The team upgrades the software that schedules that heartbeat. Something goes wrong, and the upgrade is rolled back... incorrectly. The queue of overnight work does not just pause. It corrupts. And every night it does not run, a full day of the country's money piles on top.
Days, then weeks
Wages do not arrive. Balances lie. Payments vanish into limbo. For most customers this lasts days. For Ulster Bank customers, weeks. People reportedly missed house completions. One man was reportedly held in custody because his bail payment could not be confirmed. From a scheduling tool.
What the regulators found
Regulators later fined the group fifty six million pounds, and their finding was blunt: the batch system was decades old, and the testing of that upgrade was not close to good enough for something the entire bank stood on.
The most boring system
The most boring system in the building. The one nobody demos, nobody celebrates, nobody watches. That is exactly the one that takes the whole bank down.
Every failure has a story. Every story was preventable. I'm Kevin. See you at the next one.
Sources
The notice puts the underlying cause in the backout: the team had tested backing out Version 2 to Version 1, and not Version 2A to Version 1. It does not find that the system was old. It records close to the opposite, that the group spends over one billion pounds a year maintaining its IT infrastructure, that its mainframe technology is under five years old and that it uses up to date software, and it places the fault in IT risk management, controls and testing. The Group Board's own minute, quoted in the notice, is that batch processing was taken for granted.
FCA fines RBS, NatWest and Ulster Bank Ltd £42 million for IT failures
Where the several weeks framing for all 6.5 million customers comes from, and the list of consequences the regulator established: customers unable to use online banking or get accurate balances from cash machines, unable to make timely mortgage payments, left without cash abroad, and organisations unable to meet payroll.
The other half of the fifty six million pounds, and the first penalty the Prudential Regulation Authority imposed. Both figures are after a thirty per cent settlement discount; without it they would have been sixty million and twenty million.
Settlement Agreement between the Central Bank of Ireland and Ulster Bank Ireland Limited
A separate regulator and a separate fine of 3.5 million euro, covering Ulster Bank Ireland Limited, which the two UK notices expressly exclude. It is the record that puts a number on the weeks: about 600,000 customers deprived of essential and basic banking services over a 28 day period.
Impact of Ulster Bank Systems Failure on Businesses and Consumers: Ulster Bank
The bank giving its own account to a legislature while the incident was still running, and the source for the nightly run: about 20 million transactions a day processed overnight in batches, a process it said had run without serious failure for 25 years.
RBS Bank Problems Keep Man In Cell
A syndicated reprint of a contemporaneous report, and the only record of the custody case that can still be read. The defendant was not identified, the report's word is surety rather than payment, and HM Courts and Tribunals Service is quoted saying it was aware of two cases where there was a banking issue. Its figure of 17 million customers is a contemporaneous estimate the regulators did not adopt.
Related
- 10
The Disaster That Never Happened
Two-digit years made the millennium rollover a real software risk. Years of remediation and testing helped keep the disruption limited, leaving successful prevention looking like an unnecessary warning.
- 08
Sell 1 Share... or 610,000
On 8 December 2005 a Mizuho Securities order to sell one J-Com share at 610,000 yen went out inverted, as 610,000 shares at 1 yen. The desk tried to cancel within seconds, and a defect in the Tokyo Stock Exchange's own system would not let the cancellation through.
- 04
The Bank That Sent $900 Million by Accident
Citibank meant to send about $7.8 million of interest on Revlon's loan. Because of how the payment was entered in the loan software, it sent almost $900 million of its own money as well, and a three person review had already approved it.