At the open on August 1, 2012, Knight Capital began buying and selling stocks it did not mean to own.
There was no macro view behind it. No trader had decided to get long one basket and short another. No portfolio manager had pressed the bet. The position came from an order router.
The usual shorthand is that Knight had a software glitch. Accurate, but incomplete. A glitch is a spark. Knight's problem was that the spark had market access, stale code, weak deployment controls, slow monitoring, and no fast enough way to cut the connection.
By the time it stopped, according to the SEC, Knight's automated routing system had produced more than 4 million executions in 154 stocks and traded more than 397 million shares in about 45 minutes. Knight ended up net long roughly $3.5 billion of 80 stocks and net short roughly $3.15 billion of 74 stocks.
The loss was more than $460 million.
The important number is the clock.
The old code was still there
Knight was not a small shop running a toy system. In 2011 and 2012, the SEC said its aggregate trading generally represented about 10 percent of all trading in listed U.S. equities. One of its primary systems was SMARS, an automated router that took parent orders and sent child orders into the market.
Ahead of the NYSE's new Retail Liquidity Program, scheduled to begin on August 1, 2012, Knight changed code in SMARS. The new code was supposed to replace old functionality called Power Peg. Knight had stopped using Power Peg years earlier.
The problem was that old code was still present and still callable.
The new Retail Liquidity Program code also reused a flag that had previously activated Power Peg. Knight intended to remove Power Peg so the flag would call the new code instead. Beginning July 27, 2012, the firm deployed the new code across SMARS servers in stages.
One of eight servers was missed.
The SEC order is plain on this point. One technician did not copy the new code to one server. There was no second-technician review. Nobody realized that Power Peg remained on the eighth server and the new code was not there.
The failure was ordinary: a release process missed one machine.
The market did the rest.
The machine kept filling orders that were already filled
On August 1, seven servers processed the new retail liquidity orders correctly. The eighth server did not. Orders carrying the reused flag triggered the old Power Peg code.
Power Peg had once included a function that tracked how much of a parent order had already been executed. That mattered because child orders should stop when the parent order is filled. But in 2005 Knight had moved that cumulative quantity function to an earlier point in the code sequence and had not retested Power Peg after the change.
So when the old code woke up in 2012, it did not know when to stop.
The SEC said SMARS sent child orders rapidly for each incoming parent order without regard to executions already received. Another part of Knight's order handling system knew the parent orders had been filled, but that information did not reach SMARS.
The original trigger was small: 212 incoming parent orders.
The result was not small. Millions of child orders hit the market. In 75 stocks, Knight's executions made up more than 20 percent of trading volume during the period and contributed to price moves greater than 5 percent. In 37 of those stocks, prices moved more than 10 percent and Knight's executions were more than half the trading volume.
By then it had become a market event. Other participants were trading against the error.
The warning was not a warning
There was a chance to catch it before the open.
Starting around 8:01 a.m. ET, an internal Knight system generated automated messages that referenced SMARS and identified an error described as "Power Peg disabled." The SEC said 97 of these messages went to a group of Knight personnel before the 9:30 open.
They were not treated as system alerts. Personnel generally did not review them when they arrived. The messages were caused by the code deployment failure, but they were not acted on before the market opened and were not used to diagnose the problem after the open.
The failure now moves from software to attention.
A message is not a control if nobody is responsible for it. A monitor is not a kill switch. A risk report after execution is not the same as a block before execution.
Knight had controls in parts of its system before orders reached SMARS. That detail keeps the story honest. The firm was not operating with no controls at all. The failure sat in the gap: the router that could send orders into the market did not have enough protection around its own output.
The SEC said Knight did not have sufficient controls to compare orders leaving SMARS with orders entering it. It did not have procedures to halt SMARS in response to its own abnormal activity.
The system could watch some things. It could not stop the thing that mattered fast enough.
The account filled up faster than the firm understood it
The unwanted executions accumulated in what Knight called the 33 Account. That account had a $2 million gross position limit. It was not linked to automated controls over Knight's overall financial exposure.
Knight's main risk monitoring tool, PMON, was post-execution. Senior people saw large positions building in the 33 Account after the open. But PMON did not stop orders from entering the market when limits were exceeded. It relied on human monitoring. It did not produce automated alerts about financial exposure. It also did not display the relevant limits; the viewer had to know them.
During high-volume events, PMON could be delayed and inaccurate.
People were trying to interpret live damage while the router kept adding to it.
The response made the problem worse at one point. Knight tried uninstalling the new Retail Liquidity Program code from the seven servers where it had been deployed correctly. That caused more incoming parent orders to activate Power Peg on those servers too.
The error spread from the missed server to the corrected ones.
The trader's version
Knight Capital is easy to file under "technology disaster" and leave there. That is too comfortable.
Most traders now use systems they do not fully control: broker platforms, APIs, expert advisors, order templates, bracket orders, scripts, market data feeds, exchange rules, margin systems, mobile apps. The scale is different. The dependency is familiar.
Automation is not the villain. Manual traders make bad trades slowly. Automated systems can make them quickly. The difference is the rate of damage.
If a system can place orders, the system is part of the position.
The trade includes the ticker, the thesis, and the stop. It also includes the maximum order size, daily loss limit, open order check, duplicate order check, broker-side protection, alert routing, deployment process, and the decision rule for turning the system off.
For a professional broker-dealer, some of that is regulatory infrastructure. For an individual trader, it is usually more primitive. But the question is similar: what stops the account if the operator is confused?
The worst time to invent that answer is while orders are firing.
The trade nobody put on
Knight did not set out to own a multi-billion dollar accidental book. It got there because code that should have been dead was still alive, one server missed an update, warnings were not treated as warnings, and monitoring was slower than execution.
The SEC later charged Knight with violating the market access rule and required a $12 million penalty, among other remedies. The penalty was not the hard part. The hard part had already happened in the market.
The trade nobody put on still became a real trade. Other people bought and sold against it. Prices moved. Positions settled. The loss hit capital.
Markets do not care whether risk was intentional. If the order reaches the exchange, the position is real.
Disclosure: Margin of Pain publishes research and commentary about traders, markets, and risk. This article is not investment advice or a recommendation to buy, sell, short, or hold any security, derivative, futures contract, currency, commodity, or asset.
Source trail
- SEC, SEC Charges Knight Capital With Violations of Market Access Rule, October 16, 2013.
- SEC, In the Matter of Knight Capital Americas LLC, Release No. 70694, October 16, 2013.