Nasdaq says software bug caused trading outage
A man walks past the Nasdaq MarketSite in New York's Times Square, August 23, 2013. REUTERS/Andrew Kelly
By John McCrank
NEW YORK | Thu Aug 29, 2013 4:02pm EDT
(Reuters) - Nasdaq OMX Group's (NDAQ.O) massive trading halt last week was due to a software bug and other internal technology issues triggered by problems at NYSE Euronext's (NYX.N) Arca exchange that led a key backup system to fail, the exchange operator said on Thursday.
Nasdaq said it was "deeply disappointed" by the three-hour outage on August 22, and while it pointed to connection problems between rival NYSE's electronic exchange Arca and the Nasdaq-run system that receives all traffic on quotes and orders for Nasdaq stocks, it took ultimate responsibility for the glitch.
"Our backup system did not work," Bob Greifeld, Nasdaq's chief executive, said in an interview.
"There was a bug in the system, it didn't fail over properly, and we need to work hard to make sure it doesn't happen again," he said, referring to the inability of the system to fully revert to backup mode.
New York Stock Exchange parent NYSE declined to comment.
Nasdaq said it was in the process of identifying potential design changes to make the Securities Information Processor, or SIP, more resilient, "including architectural improvements, information security, disaster recovery plans and capacity parameters."
The exchange plans to present its initial recommendations for change to the SIP governing committee, made up of U.S. exchanges and the Financial Industry Regulatory Authority, within 30 days.
Nasdaq said that on the morning of August 22, Arca connected and disconnected to the SIP more than 20 times, eating up capacity. It said the SIP's capacity was further eroded as Arca sent a stream of inaccurate stock symbols to the SIP, generating numerous rejection messages.
Each data port connecting to the SIP can handle 10,000 messages per second, Nasdaq said. But in this case, the traffic from Arca was more than double that, the exchange said.
"The confluence of these events vastly exceeded the SIP's planned capacity, which caused its failure and then revealed a latent flaw in the SIP's software code," the Nasdaq report said.
This software flaw prevented the processor's built-in backup system from resetting properly, delaying the return of data. With the system degraded, the exchange said it decided to halt trading to ensure fair market conditions.
Nasdaq said it took 30 minutes to resolve the problem and then nearly three more hours to test and evaluate scenarios to reopen the market in a fair and orderly manner.
"Nasdaq OMX is deeply disappointed in the events of August 22 and our performance is unacceptable to our members, issuers and the investing public," the exchange said. "While getting to 100 percent performance in all of our activities, including our technology is difficult - it is our objective."
(Editing by Kenneth Barry and Matthew Lewis)
