Software patch/fix Guide to Linux patch management

Tips for overcoming MSP patch management pain points

Software patch management is more difficult than it once was. Industry experts offer advice on how to best stay ahead of the patching curve and reduce risks to both MSPs and customers.

Long a part of managed services offerings, patch management today is fraught with challenges. It is no longer a perfunctory service, but a priority for MSPs. Security is the main reason, but not the only one, of course.

MSP patch management has gotten trickier in recent years due, in part, to growing complexities within cloud and on-premises environments and with the increased pace of vendor software updates. The need to update internal software like remote monitoring and management (RMM) software platforms in light of several security incidents, where hackers exploited unpatched systems to deliver malware to service prouder customers, is the main culprit, however.

"The whole point is to make sure you're plugging holes in software because most malware creators will take advantage of those vulnerabilities,'' said Gill Langston, head security nerd at RMM software provider SolarWinds MSP. Because you can't guarantee users won't click on something they shouldn't, patches help ensure an attacker can't exploit a vulnerability.

"There could be a fix a customer is waiting for … to improve their ability to work, also," he added. "At the end of the day, what's most critical is the time to remediate the vulnerability that the patches are released to fix."

But this is where a lot of MSPs run into challenges, Langston said, because service provider clients balk at having to do too many system reboots or having downtime. Also, a user often must be available to click "OK" to reboot their machine.

Gill LangstonGill Langston

MSP patch management systems, meanwhile, contain a deferment mechanism so admins can reboot machines after work hours, but some people take their laptops home. When that happens, "you have to catch stragglers the next time they start their systems, which is probably during the day when they have to get to work," he said.

Most exploited vulnerabilities are over six months old, Langston noted. This indicates a lot of systems aren't being patched in a timely manner because they are either being overlooked "or there's a hole in an IT admin's process,'' he said. So, MSPs need to deploy patches as quickly and efficiently as possible, even though people don't want to be interrupted while working.

"It's been a struggle for 12 to 13 years I've been in business,'' Langston said.

AJ SinghAJ Singh

Windows 10 feature packs, which come out about every six months on top of regular patches, are also a headache, said AJ Singh, vice president of product management at RMM provider NinjaRMM. Then there are driver updates that have to be installed on peripherals, which can cause conflict and crashes because of incompatibility due to underlying hardware issues, he said.

MSP patch management tips

End users must understand the importance of patching. This requires educating them on what software patches are meant to do, Langston said.

NinjaRMM has a centralized module that service providers can use to mass deploy patches. It also lets MSPs prioritize patches by criticality, preapprove or pre-deny patches, or make updating a manual process, Singh said.

"Any RMM tool will help [MSPs] automate schedules and make patching hands-free," Singh added. "It's up to them how they tweak and hone their model based on their target market."

Oshri MoyalOshri Moyal

Oshri Moyal, CTO at RMM software provider Atera, agreed that automated patching helps ease some of the pain. Both Moyal and Singh also suggested MSPs test patches internally on a small subset of devices first to make sure they are working correctly. "If after a week everything is OK, [MSPs] should then feel they can patch their customers' systems," Moyal said.

Other MSPs Langston has talked to deploy patches to one group of machines on the first Thursday after Patch Tuesday to make sure there are no problems. Then they'll schedule patch deployment to the remainder of their environments the following Tuesday.

"I'm a fan of that," he said. "By then, you've deployed to a test group and you've had a week's time to look for any news on a problem."

Atera always uses data in its system to evaluate how a computer performs after a certain patch has been applied. That way, according to Moyal, "we can see how the patch can cause performance issues and generate alerts to customers."

In-house patch management for MSPs

Of course, before MSPs worry about their customers, they must stay on top of patch updates for their own internal technology to avoid situations like the 2019 unpatched ConnectWise-Kaseya plugin, which was used to exploit ransomware attacks on MSPs.

"Most RMM platforms are consumed by SMBs [and] midsize businesses, and those are the most vulnerable when it comes to security and ransomware,'' Singh said. "They're more susceptible because they don't have the money to protect themselves against all these new-age threats, which could be a gateway to … their external clients."

At the end of the day, what's most critical is the time to remediate the vulnerability that the patches are released to fix.
Gill Langstonhead security nerd, SolarWinds MSP

MSPs can be so focused on supporting their customers that sometimes they forget to look inward at their own tools, Langston agreed. "You should be your best customer when it comes to keeping things updated'' since MSPs have become targets, he said. "So, patch anything you're using, and this prevents you from becoming an entry point."

Langston recommended utilizing sources like the Cybersecurity and Infrastructure Security Agency, which publishes notifications about vulnerabilities and new patches "with color on the description on how important it is and steps to take if you can't deploy a patch on what you can do in the meantime," he said.

What's ahead for patch management?

Langston said he would like to see patch management software evolve to perform predictive analysis. So, if there is an active attack underway, it sends information out on what the prime targets are to elevate the criticality of a patch in real time. "Right now, there's just an onslaught of patches, and being able to bubble up what is critical and [will give] the most bang for your buck" isn't easy, he said.

NinjaRMM has focused on making its patching engine more comprehensive. For example, it introduced a "credential store" that can run a patch scan or update a patch using a specific credential or set of user permissions, Singh said. The company is also planning to support Linux.

Looking ahead, Langston doesn't see patch management becoming any less important, calling it "one of the pillars of delivering managed services to your customers," along with remote monitoring and applying antivirus software.

Dig Deeper on MSP business strategy