AI attacks lack stealth -- nation-state actors are changing that
Today's AI attacks are still hard to miss and easy to stop, but nation-state threat actors are quietly pioneering low-and-slow adversarial AI. Learn what the shift means for defenders.
A Chinese threat actor has developed an offensive AI harness that deliberately throttles its own agents to avoid detection, while running noisy decoy attacks in parallel to distract the security operations center. That's according to Michael Freeman, senior director of threat research at ServiceNow, whose team observed the harness in the wild.
"They will compromise a building management system, and the agent will just do a couple of probes every few days -- not enough to trigger any defensive algorithms or tool sets," he told TechTarget Cybersecurity. "It's actually quite impressive."
The AI attack's low-and-slow approach makes it an alarming outlier. In Black Hat's pop-up network operations center (NOC), for example -- one of the most challenging defense environments imaginable -- managers say agentic AI attacks are still noisy, conspicuous and easy to stop. But, based on the shift in nation-state threat activity, experts say that could soon change.
"Historically, offensive techniques tend to follow a familiar path," said Jeff Pollard, analyst at Forrester Research, adding that nation-state hackers and elite researchers are generally the first to innovate, followed by advanced cybercrime groups and then criminals using commodity tooling.
"The difference this time is speed," he said. "Between the volume of research coming out of Black Hat and DEF CON, the rapid pace of frontier model development and the growing open source ecosystem, I expect AI-enabled tradecraft to commoditize much faster than previous generations of offensive techniques."
Typical offensive AI agent still 'a bull in a china shop'
For now, however, the typical offensive AI agent remains "a bull in a china shop," Freeman said -- echoing what the Black Hat NOC team reports seeing on the ground at conferences worldwide. According to James Pope, SOC lead for the Black Hat NOC, adversarial AI agents currently have the subtlety of cannon fire.
"They're not stealthy. They're trying all the things," said Pope, who is also senior director of security product research and technical marketing engineer at Corelight. "'Oh, that didn't work -- let me try this. Let me grab your credentials. Let me go put this here.' They are not low and slow right now. They're scanning and they're hitting everything."
That includes canaries, honeypots and other deception technologies, making it easy for defenders to detect and stop them.
They come in fast and loud as hell, and they just kick everything over. Right now, our greatest line of defense is the fact that they trip every wire.
Neil 'Grifter' WylerSenior network operations lead, Black Hat
"They come in fast and loud as hell, and they just kick everything over. Right now, our greatest line of defense is the fact that they trip every wire," agreed Neil "Grifter" Wyler, senior network operations lead at Black Hat and vice president of defensive services at Coalfire. By moving at machine speed, he added, malicious AI agents quickly give themselves away. "We're like, 'That's not a person; that's AI. Kill it.'"
If he were advising a red team using AI, Wyler added, he would tell them to deliberately slow down their agents -- exactly what the ServiceNow threat researchers observed a Chinese threat-actor doing in the wild.
AI harnesses could enable fast offensive gains
Freeman acknowledged that cybersecurity vendors have a vested interest in overstating the current offensive AI threat and promising that their products and services can mitigate it. Until recently, he said he was skeptical of adversarial AI's real-world capabilities.
"A few weeks [before Black Hat USA], I was going, 'This is a lot of marketing hype,'" he said. In a matter of days, however, he said he saw the rate of improvement in offensive AI shift dramatically. "The evolution is getting much, much faster as people understand that it's not the model, it's the harness that's really key."
Once threat actors understand how to develop and tune AI harnesses, they can quickly adapt and optimize their attacks. "It's just a template at that point," Freeman added. "I'm like, 'OK, I launched this attack, but I kept getting blocked at this phase. Why? What am I missing?'"
Say the harness didn't have enough information about endpoint detection and response for the AI agent to bypass it, for example. The attacker can query AI for information on EDR tools and use that info to create a new harness component. According to Freeman, such tweaks eventually result in more effective -- and therefore, lower and slower -- attacks.
'Do you feel lucky?' What CISOs should do now
For now, the typical criminal attacker is likely not well-versed in AI harness development -- just as the typical SOC likely isn't fending off sophisticated agentic AI attacks. How long that stays the case remains to be seen.
"Could vendors be hyping a problem that isn't really here yet? Sure. The problem is knowing when it will be," said Rik Turner, an analyst at Omdia, a division of Informa TechTarget. "I'd expect methodologies those folks are using in their nation-state jobs to bleed across into the commercial sphere in a very short time frame."
Could vendors be hyping a problem that isn't really here yet? Sure. The problem is knowing when it will be.
Rik TurnerAnalyst, Omdia
And there's always the possibility, Turner added, that low-and-slow AI-driven attacks are alreadyhappening in mainstream SOCs -- but with such sophistication, the targets haven't noticed.
"Attackers have always optimized against whatever defenders measure," Forrester's Pollard added. "As detection improves, expect adversaries to shift their efforts toward patience, deception and behavioral camouflage. We've seen the same pattern play out across spam, malware, ransomware and cloud intrusions. Defenders adapt. Attackers adapt. The cycle repeats."
The Black Hat NOC hasn't yet seen an AI-driven attack that meaningfully tests its defensive capabilities, but Wyler agreed that it's only a matter of time. "They will get stealthy, and that's when we're going to be like, 'well, shit,'" he said.
With unfettered access to the cybersecurity industry's top technology and talent, the Black Hat NOC is unusually well-positioned to meet that moment. The team has already deployed defensive agentic AI across multiple use cases -- and reported that its mean time to detect and mean time to contain have improved significantly in the past 24 months.
According to analysts, mainstream CISOs should follow suit.
"Learning how to operate with AI defensively now is a huge advantage. Waiting to invest just means waiting longer to learn that lesson," Pollard said. "And defensive AI can help with today's attacks -- and tomorrow's."
And, added Turner, no one wants to be the CISO whose organization is among the first to suffer a sophisticated, low-and-slow AI attack but isn't prepared.
"Not deploying defensive AI for the time being is a bit like playing Russian roulette," he said. "Or as Clint Eastwood once put it, 'You've got to ask yourself one question: "Do I feel lucky?" Well, do ya, punk?'"
Alissa Irei is senior site editor of TechTarget Cybersecurity.
