Government contractor Northrop Grumman Corp. will pay $250,000 for an independent review of the massive computer failure that cut off some state services for days last month, and the company could be pressed to pay more if the review finds its negligence led to the crash.
Northrop Grumman vice president Sam Abbate, who oversees the company's $2.4 billion, 10-year contract to provide Virginia's computer services, told the General Assembly's investigative arm Monday that the company regretted the disruption, which hampered 26 of 89 state agencies. Some services, like getting drivers licenses and paying taxes, were unavailable for a week beginning Aug. 25.
Abbate said the company was "committed to ensuring that this uncommon occurrence is carefully studied."
"As with any incident, we will listen, we will learn, we will do better," he told members of the Joint Legislative Audit and Review Commission.
Sam Nixon, head of the Virginia Information Technologies Agency overseeing the contract, said the company faces at least $100,000 in penalties. He said he also believes the state agencies should be given credit against their bill for the time that their services were unavailable.
Nixon said the state would exercise all its rights under the contract to recover financial losses.
If the review finds that Northrop Grumman was negligent the state could argue for more money, but JLARC staff said it would be difficult.
Sen. Janet D. Howell, D-Fairfax County, pressed the issue, asking staff what kinds of penalties could be imposed on Northrop Grumman.
Ashley Colvin, the commission's chief legislative analyst for VITA oversight, said the contract limited it to consequential damages only if claims result from the "willful or gross misconduct" of the company, which he said was a high bar to set.
"It's unclear if the contractual bar would be met in this instance," he said, as Howell sat back in her seat shaking her head in frustration.
The review should be finished in December.
Nixon gave legislators the first official account of what officials believe caused the failure and what happened in the days after the crash:
Around noon on Aug. 25, a data storage unit roughly the size of eight refrigerators in a data center in Chester just south of Richmond sent a message indicating that something wasn't right. An analysis showed that one of the two memory boards on the machine needed replacement.
A few hours later, a technician replaced the board and the moment it was put back into service "that's when the real outage began," Nixon said.
Officials now believe the memory board that was not replaced may have been the one that was faulty. The machine is supposed to recover on its own from such problems, but that didn't happen in this case.
Workers continued to try to fix the problem that night, but were unsuccessful. A decision was made the next day to shut down the entire storage system overnight and replace all the internal components. The system was brought back online at 2:30 a.m. Aug. 27.
Workers discovered that the failure had corrupted many of the backup databases, so they had to recover the data from magnetic tape that must be spun up and loaded into the computer, a tedious process.
Nixon likened it to knocking over a file cabinet, spilling papers from files all over the floor. All the data is still there, but "we can't find what we're looking for," he said.
Nixon said his agency is focused on figuring out how it could have improved recovery times.
"The overall outage itself - the hardware failure itself - was unacceptable," Nixon said. "The amount of time it took to restore agencies to an operational speed was unacceptable."
While 97 percent of the data was recovered, some remains lost. The Department of Motor Vehicles - one of the hardest-hit agencies - lost thousands of pictures and signatures from those trying to renew or obtain driver's licenses in the four days preceding the failure.
A Minnesota company is trying to recover thousands of those files, but officials believe at least 4,200 pictures and signatures saved on Aug. 25 appear to be unrecoverable.
Sen. R. Edward Houck, D-Spotsylvania, said he wanted to make sure Northrop Grumman covered the entire cost of the independent review.
He held up a full-page newspaper ad the company ran in major newspapers last year after a state review found that it wasn't in compliance with the contract and said the company should apologize to the public in a similar fashion.
"It seems to me that Northrop Grumman should perhaps consider making an apology in this form to the citizens of Virginia who have been disadvantaged by what has happened," he said.
Not all legislators seemed as upset.
"Accidents do happen. This was not something that was done on purpose," said Del. Johnny Joannou, D-Portsmouth. "... I just hope something like this doesn't happen again."