IT disaster at City Hall

By Sara Bruestle | May 16, 2012

The mechanical failure of several servers and backups at City Hall has resulted in the loss of important data, including financial and police.

The city’s main server and data backups failed, preventing access to its financial system, criminal evidence photos, audio recordings, grant information and other city files. The estimated total data lost is 1.5 terabytes.

Five of the city’s hard drives, nine virtual servers, and at least two backup devices were corrupted last month due to a cooling system failure in the server room. The room overheated from a cool 70 degrees to about 110 degrees, IT manager Dave Varga said.

“The cooling unit stopped,” Varga said. “Judging by how the drives failed, it leads me to believe there was a hot spot. We had some extreme heat going on. It was ridiculously hot.”

City staff has shipped the damaged hard drives to two data recovery companies in attempts to retrieve the corrupted files. No data has been recovered yet, said Finance Director Scott James, who oversees the IT department.

“I do think this is qualified as a disaster,” said Councilmember Kevin Stoltz, who is an IT expert. “This is the worst thing that could happen, to lose data and not be able to get it back and not have a backup plan.”

In the meantime, staff members are recovering what information they can from old emails, thumb drives and hard copies. They are also reviewing other backup files that can be put back into operations.

Staff has been forced to complete payroll and accounts payable manually. The most recent saved files are from December. As such, the city’s annual, mandatory state audit has been delayed.

“All the financial information is recoverable,” James said. “We are recovering general ledger entries, vendor payments and employee payroll from hard copies.”

So far, about $13,000 has been spent trying to recover the data. James estimates it will cost another $15,000 to $35,000 to get the data back.

He estimates it would cost the city another $157,000 to cover the purchase of new servers, data backups, offsite storage, maintenance, training, consultant assistance and other tools. James said the tools will offer “three layers of protection” to prevent or limit data loss in the future.

A secondary cooling unit, which would kick in if and when the cooling unit fails, is also going to be added to the mayor’s 2013 budget, Varga said.

James said he is “optimistic” that the city will recover most, if not all, of its data.

After a month, however, the first data recovery company “called it quits” and did not recover any of the data. The hard drives were then sent to a second data recovery company – which was recommended by the server manufacturer.

James said staff had decided to go with the first data recovery company and not the manufacturer’s recommendation at first, because it was local and less expensive. The decision did not go to the Mukilteo City Council.

Stoltz, who owns a computer services company, said IT management decisions should come before the council. For example, he said, staff should have taken the manufacturer’s advice.

“Once you’ve had a data loss like this, that’s the last time to cut corners,” Stoltz said.

The city recently virtualized some of its servers, which used up some of its backup storage. Staff did not replace the backups, due to budget constraints, Varga said. Virtualization creates a software version of multiple servers.

“When you start changing your whole strategy, you have to make sure you’ve accounted for all the ramifications, and backup is one of the last things you don’t want to sacrifice,” Stoltz said.

“What happened shouldn’t have happened. It was a rare occurrence, but that’s why you have different levels of backup.”

Comments (1)
Posted by: Lynn McKinney | May 25, 2012 03:27

It appears to me that the City of Mukilteo tends to take the low bidder...even when lower then their own engineer's estimate.  After years in the commercial construction/development industry, I have learned that you never take the low bidder...what did they leave out? and, you get what you pay for.

In this day and age...I agree...totally unacceptable.



If you wish to comment, please login.