The Bank That Sent $900 Million by Accident
01:25 · Point of failure
Citibank meant to send about $7.8 million of interest on Revlon's loan. Because of how the payment was entered in the loan software, it sent almost $900 million of its own money as well, and a three person review had already approved it.
Editions
- Watch
- The episode on YouTube1:56
- Read
- The 900 Million Dollar Checkboxtechnical debrief · zof.ai
Incident
- System
- Citibank's loan operations wire process for Revlon's syndicated term loan
- Date
- 11 August 2020
- Location
- New York, United States
- Toll
- Almost $900 million of Citibank's own money left the bank in place of about $7.8 million of interest. Lenders holding roughly $500 million refused to return it, and the money came back only after the appeal and a settlement in December 2022.
- Topics
- user interface · banking · controls · human factors · payments
Point of failure
Holding the principal back required three fields to be set to the internal wash account and only one of them was, and the six-eye review verified the entry rather than the amount leaving, so nothing in the process compared the payment against the intent.
Transcript
The routine payment
A bank accidentally sent nine hundred million dollars. Then a judge said the people who received it... could keep it. This is The Point of Failure, episode four.
2020. Citibank is managing a loan for Revlon, the cosmetics company. The job that day is simple: send about eight million dollars of interest to the lenders.
The checkboxes
The software they use has a quirk. To send interest only, you have to select specific checkboxes in a specific way. The team believes they have done it right.
And here is the thing. This was not one person rushing.
Maker. Checker. Approver. Three separate people review it. All three approve.
The entire loan
They press send. And instead of eight million in interest, the system wires out the entire loan. Eight hundred and ninety three million dollars. To dozens of lenders. Some of whom really did not like Citibank that week.
Citi asks for it back. Some lenders return it. Others keep about half a billion and say: no.
And because of a quirk in New York law about mistaken payments to creditors, a judge initially rules... they can keep it.
It took an appeals court, two years later, to get the money back.
What actually broke
Point of failure
All of it triggered by an interface where doing the normal thing and doing the catastrophic thing looked almost identical, and no system checked whether a nine hundred million dollar wire matched an eight million dollar intent.
Three humans said yes. The software never asked: are you sure this is what you meant?
Every failure has a story. Every story was preventable. I'm Kevin. See you in the next one.
Sources
In re Citibank August 11, 2020 Wire Transfers: Findings of Fact and Conclusions of Law
The primary record, filed 16 February 2021. Citibank, "acting in its capacity as Administrative Agent for a syndicated term loan taken out by Revlon, Inc.", intended to wire "approximately $7.8 million in interest payments to Revlon's lenders" and instead "mistakenly wired, in addition to Revlon's $7.8 million, almost $900 million of its own money as well"; the notice sent to lenders recorded that 315 of them "received their pro rata share of 893,944,008.52". It sets out the "six-eye" approval procedure, "which requires three people to review and approve a transaction before it is executed", as a maker, a checker and an approver. It also quotes the manual requiring that all of "FRONT[;] FUND[; and] PRINCIPAL" be set to the wash account, where only the PRINCIPAL field was checked, "neglecting the FRONT and FUND fields". The basis for the incident record and for the checkbox cues in the transcript.
In re: Citibank August 11, 2020
The appeal, argued 29 September 2021 and decided 8 September 2022. It records that "In total, Citibank paid out $894 million in principal and $7.8 million" in interest, that managers representing about two hundred debtholders "honored Citibank's Recall Notices and returned about $385 million", and that ten defendants "with approximately $500 million in debt refused to return the funds". Holding that "the discharge-for-value rule does not shield the beneficiary of a mistaken transfer from claims for restitution if the beneficiary is on inquiry notice of the mistake", the court vacated the district court's judgment and remanded the case. This is the appeals court the transcript refers to; the court vacated and remanded rather than ordering the money returned.
OCC Assesses $400 Million Civil Money Penalty Against Citibank
News release 2020-132 of 7 October 2020, the regulator's own record of the $400 million civil money penalty and the consent order over deficiencies in enterprise wide risk management, compliance risk management, data governance and internal controls. It backs the written companion's paragraph on the regulatory response. The release does not attribute the penalty to the August 2020 wire, and neither this episode nor the companion says that it does.
Citi settles with Revlon creditors over $500M piece of payments blunder
Report of 19 December 2022 on the settlement that ended the case after the appeal, stating that "Roughly three-quarters of the payments, which Citi testified it wired by mistake, have been returned to the bank, according to Friday's letter". This is what the published record supports behind the transcript's line about getting the money back, and behind the recovery named in the incident record.
Citibank just got a $500 million lesson in the importance of UI design
Independent report of 17 February 2021, quoting the district court on how the principal was to be directed to a "wash account", described as "an internal Citibank account... to help ensure that money does not leave the bank", and on "the confusing interface of financial software called Flexcube". It is the basis for reading this incident as a failure of interface design rather than of attention, which is the argument the written companion makes.
Related
- 02
$440 Million in 45 Minutes
A rollout reached seven of Knight Capital's eight production servers. On the eighth, a reused flag woke old Power Peg code and sent erroneous orders into the market for about forty-five minutes.
- 01
The Update That Stopped the World
On 19 July 2024, a flawed CrowdStrike content update crashed Windows machines around the world. The failure was not an attack: it was a release path that trusted a validator which had accepted the configuration it was meant to stop.