Nearly one in three organizations have already suffered a disruption caused by an AI failure, according to Forrester research — and the incidents are not minor. AI agents have deleted production environments, wiped data sources and even destroyed backups.
At this month’s Forrester Security & Risk Forum in Washington, D.C., VP and senior research director Amy DeMartine told CISOs the question isn’t whether an AI agent will fail catastrophically, but when — and whether their organization can recover when it does.
DeMartine advised organizations to look beyond AI guardrails and build the resilience capabilities needed to recover when those safeguards inevitably fail. She argued that AI resilience should be integrated into existing operational risk and resilience programs rather than treated as a separate discipline.
“Strengthen your resilience fundamentals,” she counseled. “All operational resilience mandates require that you map from your critical services down to your digital components. This is going to help you when you know an agent is hanging around and wanders through your environment.”
As AI attacks and agent-driven failures continue to escalate, DeMartine laid out a concrete playbook for what CISOs should do next — from rethinking how AI risk is categorized to building the rollback, recovery and monitoring capabilities needed to survive a worst-case scenario.
AI risks within existing risk frameworks
Organizations might be tempted to create a separate AI risk taxonomy to make emerging threats easier to identify and discuss. DeMartine strongly advised against that approach.
Instead, AI risks should be incorporated into existing enterprise risk taxonomies. AI failures can have consequences across multiple risk categories, including operational, regulatory, legal and reputational risk. Isolating AI risks makes it harder to understand and manage those broader business implications.
Organizations should also consider how AI supports critical business services. DeMartine used financial services as an example, highlighting critical activities such as payment processing, deposit management, credit assessment, investment management and customer inquiries. The level of AI resilience required depends, at least partly, on how extensively AI is involved in delivering those services.
DeMartine grouped AI use into three tiers of risk. At the lowest tier, AI acts as a productivity boost — assisting with tasks like note-taking or summarization — where a failure simply means employees continue working less efficiently. The middle tier, process acceleration, involves AI handling portions of a workflow, such as gathering information or making low-risk decisions, with humans able to intervene when needed; resilience requirements here scale with how heavily the process depends on the AI.
The highest-risk tier is process completion, where AI runs an entire process with little or no human oversight — and where organizations need to be most concerned about their ability to recover from failure.
The move from planning to action
Not every AI implementation requires the same level of resilience investment. DeMartine distinguished between scenarios where organizations can continue planning for risk and those that demand concrete action.
When AI provides recommendations, summaries or other assistance, human employees remain responsible for the work. Similarly, organizations can continue planning when AI performs an action but a human reviews every transaction.
The situation changes when human oversight becomes less immediate. For example, employees might review AI decisions in batches every two hours, rely on automated controls to identify exceptions or conduct random audits after AI has completed transactions. In these circumstances, organizations must move beyond planning and implement resilience measures.
One important capability is continuous controls monitoring. DeMartine emphasized that controls should monitor changing risk conditions, establish thresholds aligned with business risk appetite and route alerts to people who can take action. For example, an organization might establish a hard limit on how many credit accounts an AI agent can approve within a given period.
These thresholds should be measurable and enforceable. Organizations should also evaluate whether their governance, risk and compliance platforms can aggregate the necessary information and support continuous monitoring.
The AI actions creating the greatest risk
To determine where resilience improvements are most important, organizations must understand what their AI agents can do. DeMartine outlined several nightmare scenarios ranging from an agent deleting every database it has permission to access to destroying a production environment.
She asked attendees to consider an AI agent with spend authority. “A lot of people feel this is very far away, but the way things are changing, I put it on the list [of AI risks]. If an agent can spend money or tokens, then you need to be concerned, because it can spend very quickly before a human can catch it.”
These capabilities warrant close attention because their consequences can escalate before a human has time to intervene. DeMartine singled out four: the ability to delete databases, production environments or backups, which can make recovery extremely difficult; the ability to write false or malicious information into databases; the ability to modify existing data, potentially corrupting it before anyone notices; and the ability to spend — whether money or compute tokens — fast enough to create significant losses before a person can step in.
Given how quickly an agent could do real damage, DeMartine said the first step is simply knowing where the risk lives. Organizations should build an AI agent registry to track where agents operate and what systems they can access.
She warned that an inventory alone isn’t sufficient for resilience planning. Organizations should also document each agent's unique identity, owner, status and version; underlying model and associated credentials and connections; critical business services supported; and level of autonomy and permitted actions.
Recovery from worst-case AI scenarios
To strengthen resilience fundamentals and truly be ready for a significant AI failure, Forrester says organizations need several capabilities:
- Workarounds: Organizations need documented alternatives when an AI-powered process becomes unavailable. This includes trained employees, accessible procedures, offline capabilities and processes for reconciling data afterward.
- Rollback: Businesses need the ability to return systems and data to a known good state, particularly when AI agents make changes to databases or financial transactions.
- Recovery: Traditional disaster recovery capabilities remain essential, including the ability to restore databases and production environments.
- The kill switch: Organizations must be able to immediately disable an AI agent, preventing it from deleting or modifying data while IT teams attempt to restore systems.
With agentic AI attacks making headlines — including OpenAI’s frontier models breaching Hugging Face’s production servers, the first confirmed case of an AI system independently discovering and exploiting vulnerabilities without human direction — calls for a mandatory AI ‘kill switch’ have gained momentum among lawmakers and the public alike.
But as TechTarget previously reported, a single ‘off switch’ may be harder to implement than it sounds in a modern, hyper-connected tech stack, where agents can spawn sub-agents, queue jobs and call third-party tools faster than any human can intervene.
“A switch exists on paper but was never tested against an agent that’s actively working around it,” said Sophos CISO Ross McKerchar.
DeMartine’s guidance reflects that gap. She recommended that organizations test their capabilities using existing operational risk events, particularly IT failures and third-party disruptions, as inspiration for AI resilience exercises.
As organizations give AI systems greater autonomy and access to enterprise resources, the ability to detect problems, shut down agents and recover quickly will become a greater part of operational resilience.
But DeMartine doesn’t suggest waiting for the next generation of AI. “These are the muscles you have to build up before an event actually happens.”
Richard Livingston is an editor for TechTarget Cybersecurity, covering news, trends and analysis. Livingston's professional background includes editorial positions in the national defense industry covering the U.S. Army Medical Department, as well as offensive and defensive cyber strategy for military and government audiences.