City's lost data now recovered
Mukilteo has recovered almost all of the data lost after a recent crash of the city’s computer system.
The city found out recently that it will get more than 95 percent of its information back, which includes its financial system, criminal evidence photos, audio recordings, grant information and other city files.
“There is relief,” City Administrator Joe Hannan said. “Most people have been saying, ‘Let’s not have that happen again.’”
It cost the city $36,000 to retrieve the information from five failed hard drives, Finance Director Scott James said.
The city is now looking to spend an estimated $171,500 on a new system, James said.
The system failed back in April, resulting in the loss of about 1.5 terabytes of data.
At first, the damaged hard drives were sent to the company Secure Data Recovery, which has an office in Seattle. The company “called it quits” after a month, but charged the city $6,000 for trying, said James, who oversees the IT department.
The hard drives were then sent to Kroll Ontrack in Minnesota, which had been recommended by the failed system’s manufacturer, Dell.
James said staff had decided to go with Secure Data Recovery first because it was local and less expensive. They later found out, however, that the company actually has offices all over the United States.
The City Council plans to discuss the new system at the July 2 meeting, Hannan said. The meeting is scheduled for 7 p.m. at City Hall, 11930 Cyrus Way.
“That goes to relieving some of the fears of having a meltdown again,” James said. “That’s one of our biggest concerns.”
Councilmember Kevin Stoltz, also an IT expert, said an even bigger concern should be figuring out what caused the crash.
“Without identifying a path and actually what caused the other system to go, I think that’s kind of reckless to spend $170,000-$180,000,” he said. “It’s a lot of money to throw at something when you haven’t really identified what caused the problem.”
James said the city’s plans to replace and upgrade the computer system – which includes off-site backup – would help prevent or limit data loss in the future. The previous system did not have off-site backup.
For now, he said, the city is reconstructing some of its lost data, including tape and virtual backups.
Stoltz said he would like to see more options and a phased approach to replace the computer system.
“We need off-site backups,” he said. “You should never be without an off-site backup.”
The crash was triggered by a cooling system failure last summer that led to the overheating of the server room, where the city’s hard drives are stored. Five hard drives, nine virtual servers, and at least two backup devices failed on April 4.
Stoltz said a system had been set up to give email notifications to staff if drives start to fail, but “nobody was looking for the notification.”
“You have to look at your email,” he said. “That’s not a reason to go out and buy all new equipment.”
James said he, other IT staff and the city administrator would get the notifications with the new system.
Two months before the crash, the city's network engineer left for another job. The IT manager – the only other information technology staff member – had been busy working on the city’s fiber-optic line, Hannan said.
City staff had included requests for help in previous budgets, but they were not included in the mayor’s budget and were not seen by the City Council.
One request was for a temporary help-desk position that now has been approved, Hannan said. The other was for $3,000 for off-site backup, he said.
Before he left, the network engineer, who has yet to be replaced, demonstrated the city’s vulnerability by hacking into the city’s network and accessing files containing social security numbers and other information.
A temporary network engineer has now helped to secure city files with personal information, Hannan said.