$440 Million in 45 Minutes
01:33 · Point of failure
A rollout reached seven of Knight Capital's eight production servers. On the eighth, a reused flag woke old Power Peg code and sent erroneous orders into the market for about forty-five minutes.
Editions
- Watch
- The episode on YouTube1:52
- Read
- 45 Minutes: How a Dead Feature Flag Destroyed Knight Capitaltechnical debrief · zof.ai
Incident
- System
- Knight Capital's SMARS order router
- Date
- 1 August 2012
- Location
- United States equities markets
- Toll
- Approximately $440 million was lost in about 45 minutes.
- Topics
- deployment · trading systems · feature flags · dead code · controls
Point of failure
A manual rollout left one of eight servers on old code, where a repurposed flag activated Power Peg; no deployment check verified that all production servers matched.
Transcript
The missing eighth server
A company burned $440 million in 45 minutes. Not in a market crash. In a software deploy. This is The Point of Failure, episode two.
2012. Knight Capital, one of the biggest trading firms in America. At one point, their machines touched around 10% of all US stock trading. Speed is their whole business.
The night before launch day, engineers rolled new code onto their eight production servers. Seven servers get the update. One gets missed. No alarm goes off. Nothing looks wrong.
The flag and the old code
Here is the trap. The new code reused an old on/off flag. And on that one forgotten server, that flag was still wired to a piece of retired code from years earlier. Dead code. Still in the system. Waiting.
Market open
Market opens. The seven updated servers behave. The eighth wakes up the ghost. It starts firing millions of orders it was never supposed to send. Buying high, selling low, over and over, at machine speed.
For 45 minutes, no one can find the source. By the time it stops, $440 million is gone. Roughly four times what the company made the entire previous year.
Within a week, they need rescue money. Within months, the company is gone.
What actually broke
Point of failure
One missed server. One recycled flag. Zero verification that all eight machines matched. That is all it took.
Every failure has a story. Every story was preventable. I'm Kevin. See you on the next one.
Sources
The primary record. It states that Knight's trading "generally represented approximately ten percent of all trading in listed U.S. equity securities", that the new code reached seven of eight servers, that the eighth ran the retired Power Peg code behind a repurposed flag, that SMARS "routed millions of orders into the market over a 45-minute period", and that Knight "lost over $460 million" on the resulting positions. That is the Commission's figure, which is larger than the pre-tax loss Knight itself reported.
Knight's own statement of 2 August 2012: a "realized pre-tax loss of approximately $440 million" from trading out of the erroneous position, the figure the episode is named for.
Knight Capital Group — Form 8-K, 6 August 2012
The rescue money: an agreement, five days after the incident, for investors to buy $400 million of convertible preferred stock.
Net income of $115.2 million for 2011, the "entire previous year" that $440 million was roughly four times.
Knight Capital Group and GETCO Complete Merger
KCG Holdings' release of 1 July 2013 announcing "the completion of the previously announced merger whereby Knight Capital Group, Inc. ... and GETCO Holding Company, LLC ... have been combined as part of KCG Holdings, Inc., a new publicly traded holding company." The merger "was announced on December 19, 2012", within months of the incident, and KCG "will formally launch operations under its new corporate identity on July 2, 2013".
SEC Charges Knight Capital With Violations of Market Access Rule
The SEC's public summary of the enforcement action: "During the first 45 minutes after the market opened on August 1, Knight Capital's router rapidly sent more than 4 million orders into the market when attempting to fill just 212 customer orders". It also records the $12 million penalty.