Red Hat and IBM, Chainguard take on OSS security risk

Norms around CVE disclosure are not holding

Vulnerabilities are showing up faster than ever and organizations are slow to fix them, which only adds to a growing backlog.

"Log4j was a great example," Dan Lorenc, CEO of Chainguard told TechTarget. "The original reporter of that bug didn't even know it was a security issue. They just found a bug and fixed the bug publicly. Then all of a sudden, everyone realized there was a security issue and scrambled as quickly as they possibly could when they found out how bad it was. There was no kind of closed-door discussion about what to do with the handling of that issue."

Some companies cannot scramble quickly, especially larger, legacy companies. Hellekson estimated that it takes Fortune 500 companies over 40 days on average to patch a critical vulnerability.

"The number of vulnerabilities is just accumulating for every company, because they can't patch this stuff fast enough, and there's too many new vulnerabilities coming in for them to satisfy the backlog. Everybody in the industry is sitting on this backlog of vulnerabilities that they've never fixed," Hellekson told TechTarget.

AI's ability to reach in and find bugs where people don't makes the backlogs more risky.

"Some of your vendors that have been around for a longer period of time, they have larger code bases, they have much more tech debt, there's more attack surface there. And they're discovering a lot of vulnerabilities that they're understanding they're having to remediate, and they're going to have to pour the resources in to remediate that because they recognize these models are going to be out there," said Todd Thiemann, principal analyst at Omdia. "They have to get their houses in order."

Why is this specifically a problem for open source?

When a company finds bugs in software that's maintained by a vendor, the next step is simple -- ask the vendor to go solve the problem. It's less clear-cut with open source.

More than 90% of the Fortune 500 is running on open source software. Often those open source communities cannot respond to a vulnerability the same way a vendor can. There's a much wider spectrum of maintenance quality. Sometimes, critical open source projects have only a handful of unpaid maintainers -- or no maintainers.

"There's no one you can call," said Hellekson. "There's no SLA. There's no support contract. It's just stuff you pluck from the internet."

This is an issue with tens of thousands of new vulnerabilities surfacing.

"Right now, if a company needs one of those fixed, they basically only have bad options," Hellekson said.

Those options are:

1. Sink resources into fixing the issue yourself. "You could spend the money and burn the tokens to solve it just for yourself, in which case you're bearing 100% of the cost, and you are now a permanent owner of those fixes that you made, and now you have to carry those fixes over for the rest of the life of the application," Hellekson said.
2. Deal with the known vulnerability problem. "Tell the open source community, hey, I found a vulnerability in your software, which is a responsible thing to do. Unfortunately for you that also means announcing to the world that you have a security vulnerability," Hellekson said.

Project Lightwell and the clearinghouse model

The clearinghouse model that Red Hat/IBM -- and Chainguard to an extent -- are adopting aims to give organizations a third choice.

"Nobody's really done anything like this before." Hellekson said. "There's lot's of companies that will do backporting and fixes. One of the unique things about [Project Lightwell], and this is what the twenty thousand [engineers] is about, is we're not only doing that, but we are also pledging to make sure that this stuff lands in the upstream communities."

Nobody's really done anything like this before

When companies discover vulnerabilities in their own code, they can report them to a clearinghouse like Project Lightwell, which patches the vulnerability and deploys it across the software supply chain. Project Lightwell is a subscription service, letting organizations integrate fixes back into their environments. Every patch is also submitted directly back to the originating OSS project for review and acceptance.

"That's not robot work. Upstream communities already have plenty of robots sending them pull requests and patches. They're buried in those," Hellekson said, "We believe there is an essential human component to this, so that actual humans with actual relationships to the upstream communities will be able to carry and sponsor those patches back into the upstream communities."

AI vs. AI

Still, to keep up, using AI is a must for the engineers in the clearinghouse.

"Just the sheer volume means that you have to automate as much as possible," Hellekson said. "That doesn't take anything away from the human judgement stage of the process, but you need AI to review the information, do the triage, develop the remediations, test the remediations. So that's part of what we're building with Lightwell is building that platform on which we can do that."

Project Lightwell is currently working with a shortlist of design partners, including Bank of America, BNY, Citi, Goldman Sachs, JPMorganChase, Mastercard, Morgan Stanley, Royal Bank of Canada, State Street, Visa and Wells Fargo.

"What we're doing is for folks who want to buy a certain level of service, but who also understand the value of getting the fix upstream instead of just carrying it," Hellekson said. If the open source project is not interested in integrating the fix, the clearinghouse will carry the patch on behalf of the members. "We will offer the fix-up. We can't control whether they'll take it in, but we'll at least make it available."

AI has made taking on maintenance of important but unmaintained open source projects possible.

"Being able to maintain, update and act as the maintainer of last resort ... would not have been possible without armies of people in the past," Lorenc said. "AI is causing a lot of problems of scale for the industry, but I think it’s also going to make a lot of things possible."

Security maintenance of mature OSS projects with no maintainers is something Chainguard has already been working on with its EmeritOSS program, automating CVE patching alongside a small team.

Still early days, but urgency is mounting

IBM and RedHat emphasize the scale and importance of their investment in open source.

"This should be understood as big an investment as [IBM] made back in 1999 when they pledged $1 billion to Linux," said Hellekson.

Despite the size of that investment from a single partnership, groups will need to work together. Lorenc and Hellekson both mention Alpha Omega and OpenSSF as organizations to collaborate with in some way to tackle open source security challenges. OpenSSF is in the process of spinning up an open source software incident response team as well.

There's no straightforward, scalable way to notify maintainers of bugs and patch them.

"Nobody’s going to be able to find all of the bugs in all of the software on their own," Lorenc said. "I think what the world needs is a place for everyone to put their findings when you can't contact that maintainer."

Despite the uncertainty, both organizations realize the urgency in developing an ecosystem to handle the influx of vulnerabilities.

"It's just something the world needs," Lorenc said. 

Ben Lutkevich is an award-winning technology writer and editor covering IT infrastructure, app development and AI