Behind the scenes at Black Hat's network operations center

The team and its technology

When the NOC crew arrives at Mandalay Bay, it takes over the network, replacing every router, switch, firewall and access point with its own technology.

"We do that for a couple of reasons," said Neil "Grifter" Wyler, senior network operations lead at Black Hat and vice president of defensive services at Coalfire. "One is that we're control freaks. But it's also that with what happens on the network, if we need to do any type of mitigation, we can't open a support ticket with the team at Mandalay Bay and wait 90 minutes. When something hits the fan -- and it will -- we have to be able to respond immediately."

It's no easy task.

When Black Hat began more than two decades ago, the NOC consisted of Wyler and two teammates using open source scripts and hardware to secure a show with 1,500 attendees and 15 training classes that maxed out at 20 students each. This year, a team of more than 100 members uses handpicked vendor products -- and many tools of their own creation -- to secure more than 15 times the attendees and more than 100 training classes that now max out at 100 students each.

"The show has grown, and we've had to adapt and grow with it," Wyler said. "And obviously, the technologies and the demands have changed as well."

It's called the Black Hat NOC because the team's primary goal is to stand up infrastructure, said James Pope, SOC lead for the Black Hat NOC and senior director of security product research and technical marketing engineer at Corelight.

"We set up this enterprise network in a very short period of time, and then we switch roles into security. We call it a NOC because if it's not available, then there's really nothing to secure. But once it's available, then we have a lot of eyes on glass and a lot of security functions," he said.

Vendors jump at the opportunity to donate their products and NOC staff in exchange for their logo in the program or on the NOC kickboard, Wyler said, enabling the Black Hat NOC to run on pretty much an unlimited budget.

According to both Wyler and Pope, the team selects its tools with only one goal in mind: Ensure the conference has the most secure operations possible. "These are not sponsors, they are partners we choose," Wyler said. "You cannot pay your way into the SOC."

This year, Palo Alto Networks provided the firewall and an AI-enabled security operations platform that aggregates SIEM, SOAR and XDR alerts. Lumen supplied the internet, Arista the switches and access points, and Jamf the MDM software for the tablets and other devices used across the show floor. Cisco provided DNS and file analysis of the data gathered from Corelight's network visibility tools, including its NDR and threat-hunting platform.

The challenge of securing the Black Hat NOC

The Black Hat NOC is, in a word, unique. For one, the team receives pretty much unlimited resources, donated by the vendor community, so it doesn't have to worry about budget restraints. But it is also a prime target for some of the world's most elite -- and up-and-coming -- hackers. This year, someone attacked the internal servers within 15 minutes of the network being set up.

"The way we explain it to people is that most organizations looking for malicious actors in their environment are looking for a needle in a haystack," Wyler said. "Here at Black Hat, we're looking for a needle in a needle stack."

Most of the traffic traversing the network would be considered hostile in any other environment, he explained -- but not at Black Hat. After the show, Wyler told TechTarget Cybersecurity that the team received 285 million informational alerts, which it whittled down to 17.1 million threats. It then blocked 383 of those threats -- the needles in the needle stack.

The other threats are what the crew calls "Black Hat positives" -- expected behavior from the training classes held during the show and the presentations held by researchers.

Then come the outliers.

"That's the key to threat hunting. It's a game of outliers," Wyler said. For example, the NOC might detect a single threat actor trying to exploit the network. The team then zooms in on their activity and pulls available data, such as an IP or MAC address, to build a profile on that individual.

"If it is something illegal -- and it does happen, probably at this show a dozen or so times -- we go to the classroom and go, 'Doing illegal things at Black Hat is still illegal' and [the activity] usually stops," he said. If it doesn't stop, the team digs even deeper.

"We have a 100% success rate at identifying people on the network based on their traffic and where they work. We take that, tie it into the registration database and often find the individual, so we can just go, 'We can identify you, we can find you if we need to. Again, stop.' And then it stops," he said. "We don't often turn things over to law enforcement because we're like, 'You crazy kids,' and slap them on the ass and send them on their way."

AI in the NOC

AI was everywhere at Black Hat, and the NOC was no exception.

"Everybody's running around going, 'AI, AI, AI," Wyler said. "And we're like, 'That's adorable. We've been using AI for years.'"

The team began using machine learning and AI where most security teams do -- alerting -- but soon found it needed more. After evaluating several tools, however, the crew realized most couldn't handle the required loads or respond quickly enough to meet the dynamic needs of Black Hat's network. So the pros developed their own tools.

In 2024, for example, they used AI to help create FragglePacket, a Rust-based network diagnostic tool. The team had been running into network issues and no commercially available tool fit the bill.

"They just weren't doing everything -- they didn't have the features that we needed. So we're like, screw it, we'll write our own," Wyler said. During this year's show, the team enlisted AI's help to add 71 features to the tool. Now it offers attack path probing, packet fuzzing, PCAP replay, staged HTTPS analysis and a rule-based diagnosis engine. "At this point, this thing is a full-on network troubleshooting monster. It will go out and just carve through a network and find every place where things are going wrong."

The team also uses agents. Trevor, an AI chatbot interfaces with Palo Alto's security operations platform to assist with threat hunting and incident response. For example, an analyst can query Trevor about an address and it will dig through the logs for information -- saving the team a lot of time.

This year, the team introduced NOCgentic, a multiagent LLM platform that routes analyst questions to specialists, queries logs and telemetry, and responds with clear answers and next-step advice. Wyler called it a "user-friendly, hand-holdy" agent that offers level 1 threat hunters a level 3 or level 4 analyst to sit next to and work with.

The team also introduced SOCgentic, a version of NOCgentic where "the training wheels are off," according to Wyler. The team nicknamed its SOCgentic front end Postcog, short for Postcognition. It accompanies the Precogs -- sensors that alert the team to problematic behaviors on the network before they happen, a la Minority Report.

Wyler added that he and his teammates gave Postcog a comically jaded attitude. "We're working, there's a goal here, but it's also incredibly stressful," he said. "If we can make anybody half-smile, we take it as a win. Analyst burnout and the tedium [of the NOC] is real."

"[SOCgentic has] been built and hardened by this environment," Wyler said. "We built our own harnesses. We built our own skills. We trained our own models which, at the end of the show, will go up on Hugging Face." The team has since released NOCgentic on Github.

"One of the great things about [Postcog] is that you've designed a tool that works in an environment that should be well beyond anyone's worst day," said Charles Henderson, executive vice president and head of DivisionHex at Coalfire. "It's almost like having a watch for normal wear that's water-resistant to 500 meters. If I ever enter a situation where I need that and I'm not diving 500 meters underwater, I've got much greater concern than my stability."

Despite all the agents, the team said AI won't replace humans in the NOC.

"There's always a human in the loop," Wyler noted, adding that in an environment like Black Hat they need to be able to say no -- especially at an event where Black Hat positives, such as a presenter demoing an exploit, are regular occurrences.

Bart Stump, managing principal at Coalfire, said he sees AI as a force multiplier -- "a really good one," he added. "I don't want to replace the people we have. We have really smart people and just [want to] make them more efficient in what they're doing."

The future Black Hat NOC

While AI-powered threats were the talk of Mandalay, Wyler admitted they are pretty easy to spot -- for now. They are fast, he said, but also loud, not stealthy and usually end up "just kicking everything over."

"Our greatest line of defense is the fact that they trip every wire," Wyler said, explaining that AI attacks often get trapped by canaries, honeypots and other deception technologies, simply because they "don't care."

It's only a matter of time until they improve, however, which will likely be a problem for the Black Hat NOC to tackle in 2027. But with nearly unlimited resources and the latest AI tools working alongside some of the brightest minds in the biz, few teams will be better prepared.

Sharon Shea is executive editor of TechTarget Cybersecurity.