Lessons learned from Meltdown and Spectre disclosure process
During a Black Hat 2018 session, Google, Microsoft and Red Hat offered a behind-the-scenes look at the disclosure and response effort for Meltdown and Spectre.
LAS VEGAS -- When researchers discover serious vulnerabilities that affect virtually every modern computing system on the planet, how do they decide with whom to share that secret?
That was one of many challenges for the Meltdown and Spectre disclosure process, according to representatives from Google, Microsoft and Red Hat who shared their experiences during a Black Hat 2018 panel. The session, "Behind the Speculative Curtain: The True Story of Fighting Meltdown and Spectre," provided a behind-the-scenes look at the response effort and coordinated disclosure process, which stretched more than six months and included dozens of organizations.
The panel was moderated by Art Manion, senior analyst at Carnegie Mellon University's CERT Coordination Center, and it featured Matt Linton, senior security engineer and self-described "chaos specialist" for Google's incident response team; Eric Doerr, general manager of the Microsoft Security Response Center; and Christopher Robinson, principal program manager and team lead of Red Hat product security assurance.
Stumbling blocks
The Meltdown and Spectre disclosure process suffered a number of setbacks over the span of several months. The vulnerabilities were discovered, in part, by Jann Horn of Google's Project Zero, along with other parties. When the researchers officially reported Meltdown and Spectre to Intel, AMD and ARM on June 1 last year, they noted that despite Project Zero being part of the discovery teams, Google itself had not been informed of the vulnerabilities.
Despite that fact, Linton said, the chipmakers mistakenly assumed Horn had already notified the appropriate staff at Google about the vulnerabilities. As a result, Linton and his team weren't looped in on the Meltdown and Spectre disclosure until mid-July -- 45 days after the vulnerabilities were first reported.
There were more problems, according to the panelists. For example, the mitigation and patching efforts for Meltdown and Spectre didn't include web browsers until late October, when Microsoft informed the other companies involved in the coordinated disclosure effort that it had developed a proof-of-concept exploit for browsers.
The early stages of the process were also marked by limited communication and collaboration between the chipmakers, which trickled down and affected other vendors. "Surprisingly enough, the chip manufacturers don't necessarily talk to each other very frequently," Robinson said. "They don't like to share information."
Doerr said an important milestone for the process occurred in early November: Representatives from the chipmakers, research teams and a small group of affected vendors, such as Google and Microsoft, got together for a face-to-face meeting, which helped the response effort move beyond a "hub-and-spoke" system to a more collaborative system.
"It's funny how rare that kind of a meeting is," Doerr said, referring to legal and competitive concerns. "Honestly, I was blown away by the collaboration in the room. It's not that it didn't stay hard after that, but it was a turning point where the collaboration went up an order of magnitude, and I felt for the first time that we were all pulling together for a shared objective."
Doerr specifically credited Paul Turner, senior staff engineer of technical infrastructure at Google, for walking Microsoft and other vendors through his Retpoline mitigation. "Who shares mitigations with competitors? In case you hadn't noticed, Google and Microsoft don't always get along," Doerr said.
Linton agreed. "Up until then, we were all afraid to talk to one another."
Privileged information
A major dilemma that arose during the Meltdown and Spectre disclosure process was deciding who to inform before the vulnerabilities were made public and when to inform them.
Robinson said the Red Hat team wasn't officially informed of the Meltdown and Spectre vulnerabilities until the end of November, after the aforementioned face-to-face meeting.
"That was a shocking lead-in. Every computer ever made is broken; what are we going to do?" he said. "I will say that the collaboration that came before was helpful, but we still had a substantial amount of work to do."
Linton said the chipmakers "owned the embargo" for the vulnerabilities, and in accordance with responsible disclosure policy, it was their decision about which stakeholders to inform. He said, "for the most part," the chipmakers selected the right parties. But he did say he pushed for other stakeholders to be included.
Robinson echoed that sentiment. "We worked very hard to get some collaboration or at least some information sharing, because there were a couple of stakeholders that weren't included," he said. "I had to be very vocal to get other commercial Linux [companies] engaged, as well, because these folks helped us develop the fixes."
Doerr said the Meltdown and Spectre disclosure process was difficult, because so many different companies were affected, but the chipmakers couldn't risk telling everyone and then have information leak out ahead of when mitigations and patches were ready.
Although, on Jan. 3, about a week before the embargo was scheduled to lift, speculation forced Intel to publicly respond to media reports regarding rumors of the vulnerabilities.
"One of the things you've got to keep in mind when you're dealing with vulnerabilities -- and especially when it's multiparty and there's a lot of complexity -- is the more people you add, the higher the likelihood of a leak is," Doerr said. "It's honestly shocking to me that it stayed under wraps as much as it did for six-plus months, given that we had thousands of people working on this across the industry."
Linton said Google tries to use "objective criteria" for determining what stakeholders to notify about a vulnerability. And in this case, the criteria was to include OS maintainers, virtualization stack maintainers, chip designers or developers who wrote kernel drivers that interfaced with speculation in some way.
Robinson took a simpler approach to deciding which stakeholders should be included in a predisclosure process. "If you can add value, I feel you should be in there to help out," he said, adding there are different levels of disclosure, and not all stakeholders necessarily need the full details of a vulnerability.
Lessons learned
Manion asked what advice the panelists would give to vendors and security researchers based on their experiences with Meltdown and Spectre. Doerr encouraged vendors to include the researchers who discovered the vulnerabilities in the ongoing response and disclosure process. Researchers, he said, can set the tone for assessing the vulnerability and mitigating it.
Robinson said, as a member of the open source community, Red Hat gets questions from security researchers who find open source software flaws but aren't sure who to contact.
"A lot of times, we'll act as a liaison," he said. "If you've found an interesting problem and don't know where to go, talk to someone you think is related and, hopefully, they'll get you pointed the right way."
Linton said the majority of problems he's experienced with vulnerability disclosure and response can be solved with better communication. Small steps, like informing stakeholders of other parties that have been informed of a vulnerability, can be helpful. He also echoed Doerr's point about keeping researchers involved so they know why, for example, a vulnerability has been embargoed for two months or a patch has been delayed.
Who shares mitigations with competitors? In case you hadn't noticed, Google and Microsoft don't always get along.
Eric Doerrgeneral manager of the Microsoft Security Response Center
In addition, Linton cautioned the audience against turning vulnerability response and mitigation into a competition where vendors market their efforts the same way they market their products. While it wasn't a problem with Meltdown and Spectre, he said he could see it potentially becoming a problem down the road.
"You're going to start to find people writing slides about how they're awesome because they resolved this vulnerability better than anyone," he said. "Part of the rules around multiparty coordination around vulnerability sharing is, after it's over, nobody throws shade."
Ultimately, Robinson said, the experience of dealing with Meltdown and Spectre has led to improved communications between vendors in the vulnerability research and disclosure process. The shared experiences and relationships built among the panelists have made it easier to deal with new variants of Meltdown and Spectre this year.
"My experience pre-Jan. 3 and post-Jan. 3 is not night and day, but maybe dusk and dawn," he said. "We're collaborating much better now."
