Part of the damage came from the attempt to fix the problem.
In late July 2012, Knight Capital deployed new trading software to seven of its eight servers.
One server was missed.
When the software went live on 1 August, an old function that had been dormant for years activated on the eighth server and began sending unintended orders into the market.
That morning, Knight’s systems generated 97 automated emails referring to the issue.
Nobody interpreted them as an alarm.
As losses mounted, engineers removed the new code from the seven servers that had been updated correctly.
Instead of containing the incident, that action activated the old function across all eight.
Within 45 minutes, Knight had lost more than $460 million.
Within months, the company agreed to be acquired.
The SEC later found that Knight had no written procedures to guide employees through this type of incident.
That is the deeper lesson.
A crisis is the worst time to decide what the first move should be.
Under pressure, people act with incomplete information, limited time, and a strong urge to do something quickly.
Calm decisions during an incident are usually made before the incident begins.
For every critical system, teams should know:
- Who has the authority to stop operations
- What must be isolated first
- Which alerts require immediate escalation
- Where the response procedure is documented
- How to reverse a deployment safely
Decide the first move now.
Then write it where the on-shift person can find it.
If a critical system failed today, would your team know the first safe action to take?
