There’s a lot more to patching security vulnerabilities than you might think

Just because a technology company published a security update doesn’t mean the flaw it’s trying to fix is completely resolved.

A security patch release is often the beginning of what could be a months-long process for companies of all sizes that need to weigh better security with possible unintended consequences, like knocking different areas of their business offline or interrupting client connections.

While all companies wrestle with the risk surrounding patching — Look at Equifax, which analysts say will spend hundreds of millions of dollars related its 2017 breach, which could have been avoided had they patched a known vulnerability — the issue is particularly fraught for small companies. Forty-seven percent of small businesses polled by the insurer Hiscox last year said they were breached within the prior 12 months, and only 52 percent had a clearly defined security strategy. Meanwhile, nearly 60 percent of the companies polled by the Ponemon Institute said they were breached as a result of an unpatched vulnerability.

It shouldn’t be a surprise, then, that patch implementation plans vary drastically between firms, depending on the size, corporate structure and amount of money in the budget. So companies don’t always patch, sometimes deciding the vulnerability isn’t worth the potential disruption to their operations.

“Not all patches are created equal, and the first challenge people run into is how they decide what’s important to do with the time they have in a day,” said Ben Purgason, director of information security at LinkedIn.

Purgason’s group, like security teams elsewhere, analyze the severity of a vulnerability versus the likelihood it will be exploited.

“That sounds great on paper when you’ve determined the likelihood [a vulnerability] will be exploited,” he said. “But you run into a brick wall when you think about severity.”

“Severity” can mean different things to different organizations, anything from the ability to keep a website online and honor compliance obligations to its ability to protect users’ personal information.

So everyone has to figure it out for themselves, Purgason said.

How the patching plans work

Publishing company ProQuest schedules much of its work around Microsoft’s “Patch Tuesday,” the monthly rollout of software updates meant to help consumers and enterprises fend off known threats (the June version addressed 88 vulnerabilities, including four zero days).

The security team at the Michigan-based firm aims to include as many Microsoft patches as possible in its package management update software by Wednesday mornings, then spends typically three days testing those updates to ensure their reliability, said Dan Ayala, chief information security officer. If all goes according to plan, the updates are on a server before the end of the week, and on all 2,500-or-so company workstations one week later.

“Think of it as a cluster of web servers,” he said. “If there’s 15 Apache servers hosting our front end, we can take them down three at a time, and redistribute the load balance. Then, as they come back up, they’re patching.”

Ayala also has time frames scheduled at night and on weekends when patches that might affect business operations can be installed.

Heads up to hackers

ProQuest’s six-person security team had to accelerate its protocol last month when Microsoft released the patch for BlueKeep, a critical remote desktop protocol vulnerability similar to the kind that enabled the 2017 WannaCry ransomware outbreak. Even before the vulnerability had a name, Ayala says, staffers noticed there was proof-of-concept malware code that coincided with Microsoft’s first update.

That meant skipping the testing process and hoping for the best.

“There was no way we could vet the patch and get it in before the code started showing up,” he said. “So we gave an order to expedite the patching system. Places with the highest exposure we hit first, Windows 2008 and 2007 were the first to be patched, so we took the priority order and then did the rest.”

Meanwhile, ProQuest also updated its firewalls and blocked suspicious traffic patterns to reduce the potential damage. “There’s still a long tail of systems we’re getting our hands around because they’ve been offline or someone was traveling, but from hearing the first whispers of this to getting the first protections in, it was probably four hours.”

More than 970,000 systems, mostly in the U.S. and China, still are vulnerable to BlueKeep, according to numbers published by security vendor BitSight, which incorporates previous findings by researcher Robert Graham. Nearly 1.6 million systems have been patched, and the status of another 1.3 million is unclear.

Security pros at Copart, an online automotive auction company, monitor the Metasploit Project, a testbed for hacking tools, for an early warning system about which vulnerabilities they need to solve quickly.

“Maybe you have a few days once its in Metasploit until the attacks,” said Josh Danielson, Copart’s chief information security officer. “If we’re just slightly faster than the industry average [at patching] then we’re better off than 80 or 90 percent of other organizations.”

Tough problem, tight deadline

LinkedIn’s first priority its to patch internet-exposed servers accessible to outsiders, Purgason said. It’s the kind of corporate critical infrastructure that’s “extremely sensitive” because it has the potential to affect user security. A server with access to a public log-in page, for instance, is charged with determining whether a potentially malicious outsider can access the site.

“Our severity goes through the roof when it affects our ability to keep members safe,” he said.

For now, Purgason added, he’s focused on operating system patches and the attack known as ZombieLoad. That architectural flaw could undercut an organization’s security by introducing vulnerabilities in technologies that were once thought automatically safe, he said. The issue could leak information and allow outsiders to improperly read sensitive data, though fixing altogether could slow patched technology by up to 40 percent, Apple said.

“You have a situation where you could have a very severe impact,” Purgason said. “Various companies have done testing to find that you could have double-digit performance degradation if you completely patch for Zombie Load, so you’ve got a problem on two fronts.”

And there isn’t always a lot of time to make those decisions.

Copart aims to install its patches on a 48 hour schedule, a big jump from the monthly maintenance windows the IT team works on now. The complexity of that process may mean Copart never hits that 48 hour goal, Danielson says, but accelerating the company’s patch implementation to bi-monthly, then speed it up again, only means more security.

Such a strategy requires more efficient automation, he says, so the company lately has automated its quality assurance, reboots, digital checkups and other processes. It’s all in response to recent pressure from higher-ups invigorated by a possible $17 billion market cap, he says.

“We’re working on it now,” he added.