All systems aren't go at City Hall
Computer system failures at City Hall go beyond the loss of important data, according to two former city employees.
Other shortcomings in the system could also mean dangerous security breaches, one of them warned.
Last month, the city’s main server and data backups failed, resulting in the loss of its financial system, criminal evidence photos, audio recordings, grant information and other city files. The estimated total data lost is 1.5 terabytes.
However, according to the city’s former network engineer and former accounting services manager, the underlying issue in the city’s IT (Information Technology) department is poor management and poor decision-making.
They said it’s an issue in other departments, too, and that they quit their jobs because of it.
“There is no excuse, in this day and age, for a city to lose all of their data like this,” said Gini Schacker, who quit three weeks after the servers crashed. “The fact that they allowed this to happen is unacceptable.
“In my opinion, heads should roll. I stand by that comment.”
Josh Spradley resigned as the network engineer two months before the servers failed. However, he said a disaster like that was “imminent” due to mismanagement of the IT department.
“I’ll never consider working for local government again,” Spradley said. “I was completely poisoned by local government.”
When he quit, Spradley said the city had many data security vulnerabilities, including unprotected employee and business license data.
As the network engineer, he had hacked into all of the network devices, decrypted passwords and accessed files containing social security numbers and other personal information. He didn’t receive any authorization to fix the problem.
“I don’t know what’s not susceptible to hacking, if you’re an expert in IT,” said Finance Director Scott James, who oversees the IT department.
“Security is an ongoing battle. We do have some appliances to limit intrusions in our system, but I can’t say they’re going to protect us completely.”
Spradley said that when he tried to tell the city administration about holes in the network, they didn’t want to hear it.
“They didn’t want to know anything about the network,” Spradley said. “Security, like IT in general, at the city of Mukilteo is an afterthought.”
Both ex-employees said the IT manager – Dave Varga – is a manager only in title, and that the administration routinely makes technology-related decisions.
“The IT manager there has no power,” Spradley said. “It’s all in the directors’ hands, and they make very uninformed decisions based solely on money.”
City Administrator Joe Hannan said their stories are “inaccurate.” Although “it’s not unusual that employees disagree with their boss,” Hannan said that staff is following Varga’s advice – for example, the IT proposal – and some of Spradley’s, too.
“I don’t think the record says that, in Dave’s case, that he’s not given both the authority and the ability to make some changes,” Hannan said.
James said there is “some truth” to Schacker and Spradley’s complaints, but that theirs is only one side of the story.
“There were some shortcomings in IT,” James said.
He said staff tries to find a “good balance” between cost and effectiveness when making IT decisions.
“When we’re doing the decision making, budget is always a factor,” James said. “It’s a fact that we’re spending more than we’re taking in. When we can’t fund one option, we look for what other options there are.”
Five hard drives, nine virtual servers, and at least two backup devices were corrupted due to a cooling system failure in the server room that occurred last year.
Varga said the room overheated to about 110 degrees more than six months ago, which led to the failure of the hard drives last month. Even so, Varga said no one could have predicted when the hard drives would fail.
“Excessive heat will decrease lifespan, but as far as predictability, you can’t predict when a drive will fail,” he said. “There’s no way to know it will fail until it does.”
City staff has shipped the damaged hard drives to two data recovery companies in attempts to retrieve the corrupted files. No data has been recovered yet, James said.
A proposal to replace and upgrade the city’s technology infrastructure and disaster recovery systems is estimated at $171,500.
James said the tools would offer “three layers of protection” to prevent or limit data loss in the future.
For now, the city is reconstructing some of its lost data, including tape and virtual backups.
Varga said there hasn’t been much of a budget for security because there hasn’t been awareness for it. He said he’s hopeful that will change.
“For government, this is pretty important stuff,” he said. “We want to get this as tight as possible. We’ve run into a lot of battles, but the next step is awareness.”
“How critical our data is is right in front right now. It’s pretty darn critical that we make sure it’s protected.”
Before he left, Spradley helped Varga increase network security, including a more advanced firewall.
Varga said the IT department is looking to hire a company to test its network security in the near future. The last time a consultant tested its security was in 2000, James said.
“We’ve closed a lot of gaps and improved network security here, but there is still stuff that needs to be done,” Varga said. “There are always things that can be tightened up, but we try to tighten up as good as we can.”