AI agent security: How reliable is enterprise AI testing?
Experts outline comprehensive testing practices before and after deployment of AI agents to prevent them from acting on their own beyond prescribed instructions.
As AI systems evolve from chatbots into autonomous agents capable of making decisions and taking actions, their security and reliability have been called into question. Businesses should address those concerns during the testing phase of AI agents before they're deployed and allowed access to corporate data, tools and systems. But how adequate is enterprise AI testing?
Enterprises might have to rethink how they test AI systems, according to a routinecybersecurity evaluation of frontier AI agents by the UK AI Security Institute (AISI), which documented several instances of unpredictable agent behavior. AISI helps governments better understand the risks posed by advanced AI, including evaluations of frontier AI models for capabilities in areas such as cybersecurity.
During one of its cyber evaluations in July, AISI found AI agents were taking actions outside the intended scope of the tests, including actions involving real people and organizations. The institute conducted 122 evaluation runs across several AI models and found unsanctioned activity in 10 of them. Across those 10 evaluations, there were 19 instances in which an agent took rogue action; 17 of them involved Anthropic's Mythos 5. The other two occurred during a single run with OpenAI's GPT-5.6 Sol.
The most serious rogue behavior involved a Mythos 5 agent tasked with solving a cybersecurity challenge. In pursuit of its goal, the agent attempted a supply chain attack on an open source software project. It researched the project's maintainers, created fake identities and tried to socially engineer a human reviewer into approving malicious code. When challenged, the agent attempted to cover up its previous activity by making it appear harmless and considered using another identity to continue.
The agent acted on its own and wasn't instructed to deceive anyone. The deception occurred as the agent persistently pursued the objective it was given, according to AISI. "This is the first time AISI has seen deception of this severity that was targeted at a real person, unprompted, in the real world," the institute wrote in its findings.
AISI said misconfigured task prompts contributed to some of the behavior by leading agents to incorrectly believe they couldn't complete the challenge within the intended parameters. But that didn't fully explain the findings. In some runs, agents took unsanctioned actions even when they had the instructions needed to complete the task as intended.
The agents' attempts at deception were unsuccessful. AISI said its investigation found no resulting real-world harm. Still, the institute classified what happened as a serious security incident.
Where enterprise AI testing falls short
The AISI's findings raise a significant question for enterprises preparing to give AI agents greater autonomy and access to corporate systems: Is testing an AI system's ability to successfully complete a task enough, or must organizations also test what the system might do in pursuit of that goal?
The AISI findings will encourage enterprises to be more cautious when evaluating and deploying agentic AI solutions.
Lian Jye SuChief analyst, Omdia
For businesses beginning to deploy autonomous AI agents, the AISI's findings are highly significant, said Lian Jye Su, chief analyst at Omdia, a division of Informa TechTarget. But he cautioned against drawing a direct comparison between the evaluation and how these systems would typically operate in an enterprise environment.
The agents were removed from cybersecurity guardrails and filters during the evaluation, which isn't how they're provided to enterprises, Su said. Still, many businesses lack the AI engineering and governance expertise needed to implement agentic AI and rely heavily on vendors and partners. "The AISI findings will encourage enterprises to be more cautious when evaluating and deploying agentic AI solutions," he noted.
The institute's findings also expose shortcomings in how organizations currently test AI systems, Su added. Many enterprises still rely on short-horizon evaluations and human oversight, approaches that become less effective as agents gain the ability to develop complex plans, use tools and access external systems. Testing, therefore, must include a multistep evaluation of an agent's behavior under different conditions, Su advised, to account for the possibility that the agent might act beyond its intended scope.
The testing and deployment gap
The biggest mistake CIOs and CISOs can make is treating AI agents like traditional software without accounting for their increasing reasoning and planning capabilities, said Dennis Xu, research vice president at Gartner. AI agents are adaptive and can do things traditional software wouldn't do when completing a task, he explained.
Those differences require security teams to test for vulnerabilities and behaviors specific to AI agents. That includes determining whether an agent can be jailbroken by provider restrictions or otherwise manipulated to leak sensitive information or misuse tools it has legitimate permission to access. Businesses should also test whether agents can be tricked into using legitimate internet access in harmful ways -- for example, downloading malicious instructions that could lead to additional damage, Xu explained.
The risks extend beyond whether an individual agent can be compromised. As AI systems' reasoning and planning capabilities increase, they could potentially "cause far more damage, at a higher velocity than traditional software or human attackers," Xu warned.
In addition, AI testing must extend beyond vulnerabilities in the underlying systems to the environment around the agent, including its network access, permissions, credentials, tools and external connections, said Cliff Steinhauer, director of information security and engagement at the National Cybersecurity Alliance.
I think organizations are buying intelligence faster than they are designing the authority around it.
Randall HuntCTO, Caylent
"The key is that instruction is not containment," he said. "If an agent isn't supposed to access the internet or reach a particular system, that restriction needs to be technically enforced. We shouldn't rely on the model to enforce its own boundaries."
The gap between more comprehensive testing and deployment becomes increasingly problematic as agents gain permission to take actions rather than simply generate responses. "I think organizations are buying intelligence faster than they are designing the authority around it," said Randall Hunt, CTO at IT consultancy Caylent. "Many teams test an agent on the happy path, but far fewer test what it does with poisoned context, ambiguous instructions, tool failures, loops of retries, unintentional permission escalations, attackers hiding instructions in documents, and on and on."
Before allowing an agent to write and commit code, for example, organizations should test malicious package suggestions, secret handling, data exfiltration, destructive commands, prompt injections embedded in repository files, runaway retries and rollback, Hunt said. How much freedom to think and act an agent receives should depend on its potential blast radius, the reversibility of its actions and how observable its behavior is. Businesses can deploy agents with read-only access before gradually allowing narrow write permissions with transaction or write limits and explicit human approval requirements. Actions that move money, delete data, change identity and access management settings, expose sensitive information or communicate externally often warrant greater scrutiny, Hunt said.
Su also emphasized the role of simulating a cyberattack before deployment. Given the AISI's findings, red teaming should specifically target deception, social engineering, creative workarounds, tool exploitation and coordination among agents, he added.
AI agent testing beyond deployment
More extensive pre-deployment testing won't anticipate every way an autonomous agent might behave, especially as the AI models, tools and data an agent relies on continue to change. Therefore, businesses should implement observability measures, including LLM tracing and tool invocation monitoring, along with continuous evaluation and red teaming in production-like staging environments, Su advised. "Enterprises need to treat the full agent system as the unit of evaluation and risk," he added.
Models, prompts, tools, dependencies, data and attacker behavior can all change, Hunt said. Enterprises need to conduct pre-deployment evaluations combined with continuous testing against production traces, runtime monitoring and periodic penetration testing or red teaming. Monitoring should also include attempted and failed actions, not just what an agent successfully completes. Tracking attempted and failed actions over time, he noted, could reveal how agents explore their boundaries, giving security teams information to inform future testing and red teaming.
Governance must keep pace with agent autonomy
Testing alone won't address the risks of increasingly autonomous agents. Enterprises also need controls governing where agents operate, what they can access and what actions they're permitted to take.
"Access to tools and the internet must be clearly outlined and made transparent," Su said. Additionally, agents should operate in sandboxed environments with real-time monitoring and human oversight designed to limit malicious activity and its potential effects.
If you move too fast from a controlled pilot to broad production access, you're going to have a bad day.
Cliff SteinhauerDirector of information security and engagement, National Cybersecurity Alliance
Governance begins with knowing which agents are operating within the enterprise. Businesses should introduce processes that discover agents, identify misconfigurations that create security risks and establish guardrails defining which actions an agent is allowed to take, Xu said. Enterprises also need what he described as rogue agent management -- mechanisms that can block malicious or risky actions an agent attempts.
Steinhauer similarly cautioned against giving agents broad access as they move from pilots into production. He recommended a "crawl, walk, run mentality," beginning with a narrowly defined scope and expanding an agent's access only after organizations have tested the controls around it. "If you move too fast from a controlled pilot to broad production access, you're going to have a bad day," he warned.
Implementing additional layers of controls around agents is another good practice. Each agent should have its own identity and short-lived credentials, Hunt suggested, and businesses should enforce least-privilege access and tool allow lists, along with instituting policy checks, network and data isolation, spending and rate limits, protected execution sandboxes and immutable logs that allow businesses to reconstruct an agent's actions. Enterprises should also monitor unusual sequences of tool use, privileged executions, repeated failures and behavior that begins to deviate from an agent's assigned task.
And when those controls fail, businesses need a kill switch, Hunt said. "You need a switch you can pull to shut it all down if things get out of control. I think organizations are buying intelligence faster than they are designing the authority around it."
Liz Hughes is an award-winning editor and writer covering AI and emerging technology and the former editor of AI Business and IoT World Today.
