The risks of failed patch management
Some risks -- like security vulnerabilities and system downtime -- are obvious, others not so much. Good patch management also requires weighing the possible risks of patching.
Joshua Skeens recalls pushing out a firewall update a few years ago that caused a critical issue for a customer. "We had to scramble and roll it back," said Skeens, COO of Logically, a managed service provider. Nine times out of 10, he said, patches work fine and don't cause a problem.
Although the odds are slim, patches can still wreak havoc. "No vendor can have access to every environment," Skeens said. "They can test it for weeks or months, but every customer's environment is different, and there could be something there that the patch doesn't play nicely with."
For Skeen's customer, the issue was content filtering. Unbeknownst to everyone, the patch changed a flag inside the operating system that changed the default in the content filtering engine. That blocked all internet traffic to about 300 of the client's stores, which were down for as long as 12 hours in some locations.
What is patch management?
Patch management is both a process and software that organizations use to automate vendor updates and fix critical flaws in applications and operating systems.
"We always see patch management as good hygiene," said Cortney Thompson, CIO of Lunavi, another managed service provider (MSP). Patch management has long been a standard service offered by MSPs.
Most companies understand the need to stay current with their software and keep patching up to date to decrease the risk of cyber attacks, he said. Now they're looking for the most optimal way to execute patch management.
But Skeens said many companies don't have processes in place to look proactively at their systems and decide when to patch them. Specialized patch management software is a common solution to the problem.
Some tools will know when new devices are added to the network, a capability that has become more important with the sharp rise in remote work during the COVID-19 pandemic. There are patch management tools that let MSPs or corporate IT departments install agents on machines to see what application software and operating systems they are running and whether updates are being applied. They also provide the ability to schedule patches across the board rather than in a one-off scenario, Skeens said. "Not enough [companies] are doing this," he said.
What could cause patch management to fail
From incompatible hardware to conflicts with another patch, or a patch that installs well but breaks something else, things can and do go wrong, according to Matthew Hodson, co-founder and CIO of the MSP Valeo Networks.
"It's very hard to be 100% patched because there are always new patches coming out and often there are conflicts with other systems," Hodson said.
The reason patch management fails can be something as simple as a user's laptop not being turned on, he said. Another common issue is that companies lack the expertise or manpower to do patch management correctly. "A lot do 'set it and forget it,' and that's not good," Hodson said. "You need a report if something failed and why."
A patch can also fail if there is activity going on with a computer that prevents it from installing, or, as Skeens discovered, the existing software somehow conflicts with the patch, especially if it is custom designed.
Weighing the risks
Handling patch management risks often requires evaluating the risk of deploying a patch against the risk of not deploying it. Hodson shared an anecdote that illustrates why.
Microsoft puts out monthly patches on "Patch Tuesday," but some patches must be deployed quickly to address certain vulnerabilities, such as zero-day threats or exploits, which are discovered by hackers before the software vendor is aware of them. In the latter instance, Hodson said, patching is an all-hands-on-deck affair.
"It's like a firestorm because it's out in the news, everybody's scrambling to install the patch and bad actors are trying to get in," he said. "We push those as soon as possible, depending on the criticality."
He recalled a zero-day vulnerability a couple of years ago that was related to a printer exploit. "We had a discussion with some key clients, and one said, 'We can't not print,' and the vulnerability was low, so they decided not to patch," even though 99% of the time with zero-day exploits Valeo pushes clients to patch, he said.
Eventually, Microsoft rewrote the patch so it wouldn't affect the company's printing abilities, Hodson said.
The problem with zero-day patches is there isn't a lot of time to test them to ensure they won't interfere with other systems. "There's a gamble: Release and install quickly, and you risk breaking something that's not truly tested," he said.
Risks of failed patch management
Patch management pitfalls include pushing out updates too quickly and devices going offline. But the most significant risk when patch management doesn't take, not surprisingly, is leaving a system vulnerable to malicious actors.
"Like the Equifax breach demonstrated, if you're not paying attention to what software is on your machines and making sure updates are being applied, you're leaving yourself vulnerable to hackers," Skeens said. He likened it to leaving a door to a house unlocked, giving someone the ability to easily come inside.
Another risk of failed patch management is software that is out of date, which increases its vulnerability, Thompson said. "Sometimes it's a race between bad actors out there who want to take advantage of these vulnerabilities or bugs and system administrators trying to patch in a timely manner."
Patch vulnerability management best practices
These experts all agree that it's important to have a patch management process that specifies how patches will be applied and what type of testing will be done, so you can use that information as a point of reference. "Don't do willy-nilly installation of patches," Hodson said.
The MSPs also agree that patches should be tested on a small subset of machines. Hodson said Valeo Networks installs patches on its internal systems first, monitors them, and then staggers them out to its customer base. Then it provides customers a report of the results.
"You don't want to be one of those companies that pushes a patch out to the entire company and have it fail," Skeens said.
A patch management system should also provide good visibility, reporting and alerting, Thompson said. Lunavi uses "layers" of tools that are mostly automated and have auditing and reporting capabilities, to ensure that IT knows about patches that were not executed, he said.
When it tests patches, Lunavi also takes snapshots to look for any inconsistencies or issues so IT can go back to a particular point in time and revert to a system state that is known to work. "Once we know a patch has been successful, we delete the snapshots, therefore permanently committing the change to the system," he said.
Hodson advised making sure to follow up on why a patch failed. "That's the biggest piece. Don't set it and forget it. A lot of companies that aren't mature in IT don't know when the last time was that they patched and that only a percentage of them were successful. They don't take the time to review the data."
