When AI agents go rogue: What enterprise leaders need to know

The risk isn't just rogue behavior

These recent incidents are notable because the models behaved unexpectedly and circumvented safeguards intended to limit what they could access and do.

An agent can make one wrong decision and then make the situation worse by taking additional actions based on that first mistake.

The models were being evaluated by some of the world's leading AI companies in environments specifically designed to test their capabilities while keeping them contained. Sandboxes and other containment measures are used to isolate models from the internet, external systems and other resources they don't need to complete a task.

Yet multiple systems found ways to reach beyond those boundaries. In several cases, the issue wasn't a malicious objective but an unintended opening, such as internet access or a configuration error, that the model was able to exploit.

For example, OpenAI's models were attempting to solve a cybersecurity challenge when they exploited a previously unknown vulnerability to get outside their sandbox and reach Hugging Face. Kimi K3 used an unintended internet connection to search GitHub for answers rather than solving its assigned challenge.

That distinction matters for companies deploying AI agents. An agent doesn't have to become intentionally malicious to create a business problem. It could simply pursue a legitimate goal in a way its developers or users didn't anticipate.

"An agent can make one wrong decision and then make the situation worse by taking additional actions based on that first mistake," said Sudhakavya Bodapati Venkata, a DevOps engineer at Starkey Hearing, a digital hearing technology company and author of a paper on how LLM-driven incident-response systems can compound risk through repeated remediation actions.

Once an agent has access to data, tools, identities and external systems, the blast radius of a mistake can increase significantly.

Venkata described a scenario in which an agent trying to restore an unhealthy service might restart it, increase capacity, change traffic routing and roll back a deployment. Each step might seem reasonable on its own, she said, but together they could create new problems, increase costs or disrupt other systems.

"The danger comes when the agent uses the results of its own actions to decide what to do next," Venkata said. "It can keep taking action even as the situation gets worse."

That ability to act on its own decisions is what can turn a model error into a real-world incident.

"Autonomy turns a model risk into an operational risk," said Naynesh Patel, secure and responsible AI and data protection lead at Accenture, a global professional services and technology company. "A model that produces a bad answer is a quality problem; an agent that can send email, change a record, call an API or spend money can turn a bad decision into a business incident," he said.

Access determines the blast radius

The surrounding environment, including an agent's permissions, tools, network access and connections to other systems, determines how much real-world impact it can have.

"Once an agent has access to data, tools, identities and external systems, the blast radius of a mistake can increase significantly," said Dane Sherrets, staff innovations architect at HackerOne, a cybersecurity company.

An AI agent with access to a CRM could potentially modify customer records. One connected to a code repository could change software. An agent with access to financial systems could initiate transactions, while one connected to internal communications could send messages on an employee's behalf.

The more systems an agent can access, the greater the potential consequences of a mistake.

This makes least-privilege access especially important for agent deployments. Rather than giving an agent broad access to an application or database, companies can limit it to the information and functions it actually needs. But explicit permissions aren't the whole picture. An agent might also inherit permissions through the systems it's connected to, creating access that's broader than it needs.

Companies also need to account for the possibility that several individually low-risk permissions could be chained together to create a more serious outcome, Sherrets said. "Read access here plus write access there can create an outcome that nobody explicitly approved," he said.

Companies need to consider not just whether an agent can access a system, but what it can change, send, execute or approve once it's inside.

For lower-risk tasks, an agent can operate independently. For actions involving sensitive information, production systems, money or external communications, businesses might require human approval before the agent can proceed.

"The line for human approval should be based primarily on consequence and reversibility," said Yogesh Badwe, CSO at Druva, a company that provides cloud-native data protection, backup and cyber recovery. "If an agent is performing a bounded, observable action that can be easily reversed, requiring a human in the loop defeats much of the value of automation."

The goal isn't necessarily to keep a human involved in every action. Instead, organizations should determine where an agent's autonomy ends based on the potential consequences of its actions. Businesses should consciously decide those boundaries rather than letting permissions make that decision by default.

Testing the model isn't enough

Even the right permissions and safeguards don't guarantee that an agent will behave as expected. The model is only one part of the system. An agent's behavior can depend on the tools it can call, the data it receives, the instructions surrounding it and the environment in which it operates.

Testing the model alone isn't enough. Companies need to test the deployed system: model, prompts, memory, tools, identity, permissions, data, network paths, orchestration logic and human-approval process.

The Anthropic, Meta and Kimi K3 incidents all illustrate that point. The Anthropic and Meta models accessed external systems during cybersecurity evaluations, with problems in the testing environment contributing to how that access became possible. In Kimi K3's case, a network misconfiguration gave the model unintended internet access, which it used to search for answers.

For enterprises, testing a model in isolation doesn't necessarily reveal how it will behave once it's connected to business tools and data.

"Testing the model alone isn't enough," Accenture's Patel said. "Companies need to test the deployed system: model, prompts, memory, tools, identity, permissions, data, network paths, orchestration logic and human-approval process."

That also means security teams might need to test scenarios that go beyond conventional model evaluations, including the following situations:

• An agent receives malicious instructions through a document.
• A tool returns unexpected information.
• An agent discovers it can accomplish a task faster by taking an action that wasn't explicitly prohibited.

Organizations should also test how an agent behaves when its actions begin influencing the evidence it uses to make subsequent decisions. Starkey Hearing's Venkata recommended controls such as action allowlists, blast-radius limits, short-lived credentials, canary testing, tested rollback procedures and an independent kill switch.

"The agent should never control or modify the mechanism responsible for monitoring and stopping it," Venkata said.

And testing is only part of the strategy. Organizations also need to know when an agent is behaving differently from what they expect. In two of the three incidents Anthropic disclosed, the affected companies didn't detect the unauthorized activity themselves. Anthropic discovered the behavior during its own evaluation process and contacted them afterward.

That raises another question for enterprises: If an agent starts accessing systems or taking actions outside its intended role, can existing monitoring tools detect it?

Traditional access logs can show which account accessed a system and when, but they might not reveal what prompted an autonomous agent to take the action, whether it was necessary to complete its task or whether the action was outside its intended scope.

"Monitoring shouldn't just be a log you check after something goes wrong," HackerOne's Sherrets said. "It should be active and continuous, watching for drift between what an agent is supposed to be doing and what it's actually doing."

That could include monitoring tool calls, access patterns and behavior that deviates from the task an agent was assigned. For example, Sherrets said organizations could alert a human when an agent calls a tool or domain that isn't on an approved list, while a monitoring agent could flag behavior that appears abnormal.

Businesses might increasingly need AI agents to monitor other AI agents because humans alone likely won't be able to keep pace with the volume and speed of agent activity, Sherrets added. Enterprises need greater visibility into what agents are doing and why, not simply which systems they can access.

Monitoring what agents can do

Most governance frameworks weren't designed to address the risks created by autonomous agents. Many enterprise AI policies have focused on what AI produces, including biased outputs, inaccurate information, sensitive content or inappropriate recommendations. But agents introduce a new layer of risk in addition to generating outputs, they can take actions.

An AI agent can do exactly what it was authorized to do and still create a serious security or operational incident.

That means governance programs need to define more than which AI tools employees can use. They also need to establish who owns an agent, what systems it can access, what actions it can take independently and which decisions require human approval. Governance as code can help companies turn those policies into enforceable controls for AI agents.

"Governance has to move from reviewing AI outputs to governing delegated authority," Patel said. "For each agent, the business should be able to answer four questions: Who owns it, what is it allowed to do, what evidence shows that it did it and who can stop it."

That authority needs to be defined carefully because having permission to take an action doesn't mean the action will produce an acceptable outcome.

"An AI agent can do exactly what it was authorized to do and still create a serious security or operational incident," said Druva's Badwe. Even with legitimate permissions, an agent could alter a configuration, delete data, change access and disrupt a workflow without anything being technically compromised.

Those boundaries should be revisited whenever an agent gains new capabilities or access. For example, an agent that starts with access to a single database might later be connected to a second application or given additional tools. Each change can alter the system's risk profile. Governance shouldn't treat approval as a one-time event at deployment; rather, it must follow the system through its lifecycle.

The same principle applies to third-party vendors and integrations. An agent's risk can change when it connects to external tools, services or infrastructure, so security assessments should extend beyond the model provider itself. Vendor assessments might need to examine how AI agents are isolated, monitored and controlled across the entire technology chain, including plugins, data connectors and other tools an agent can use.

Identity and access controls also must account for how agents behave differently from traditional software. Many existing systems were designed for human users or software that behaves in predictable ways, Sherrets said. AI agents can decide how to complete a task, making it harder to predict exactly how they will use the access they're given.

Ultimately, governance needs to account not just for what an agent is authorized to access, but how it can use that access as its capabilities and connections evolve.

Prepare for when an agent goes wrong

Even with careful governance, testing, permissions and monitoring, businesses can't assume an agent will behave as expected. Companies need a plan for what happens when it doesn't.

That could include the ability to revoke an agent's credentials, disconnect it from specific systems or stop an automated workflow. IT leaders should know who has the authority to take those actions, how quickly they can be carried out and what records are available to determine what the agent did.

The goal isn't zero autonomy. It's bounded autonomy: the agent can act quickly inside a clearly defined envelope, while the organization can see, interrupt and recover from behavior outside that envelope.

But stopping the agent is only the first step. Organizations also must understand what happened before the agent was stopped and how to recover from the changes it has already made.

"Containment is important, but recoverability is just as important," Badwe said. "It requires visibility into identity activity, API calls, configuration changes, data movement and downstream systems."

Businesses also need a reliable record of the environment, he said, to distinguish legitimate changes from harmful ones and return affected systems and data to a known-good state.

The scale of an agent's actions also matters. An agent that repeats the same action across thousands of customer records or production systems can turn a small mistake into a much larger incident.

That's a limitation of least-privilege access in an agentic environment: It can restrict what an agent is allowed to do, but not necessarily how quickly or repeatedly it can do it. For example, a tightly scoped agent with permission to make outbound payments could still make payments to the wrong destination or in the wrong amount.

Organizations therefore need to consider autonomy alongside containment and recovery. Before giving an agent meaningful authority over a business-critical system, they should know what actions it can take, how those actions can be stopped and how the business will recover if something goes wrong.

"The goal isn't zero autonomy," Patel said. "It's bounded autonomy: The agent can act fast inside a clearly defined envelope, while the organization can see, interrupt and recover from behavior outside that envelope."

The recent incidents don't show that AI agents are inherently unsafe. They show that autonomy, access, monitoring and containment must be considered together. As these systems become more capable, enterprises should plan for unexpected behavior rather than assume their safeguards will always prevent it.

Kinza Yasar is a technical writer for Informa TechTarget's AI and Emerging Tech group and has a background in computer networking.