Controlling agents a joint priority for data, AI governance teams

Setting limits on what agents can do

Siddarth Jain, AI engineering lead for finance and supply chain at OpenAI, said during a presentation that limits on permissions and access to data sources must be built into agents, especially ones that run background processes without direct human oversight. It isn't a static process, he added. For example, automated controls should continuously check an agent's permissions as it runs jobs in case they change.

Jain said auditable evidence of what agents do in business systems should also be documented -- and then reviewed by humans, at least in some cases. "I'm not saying you need to have restrictive controls everywhere, but it depends on what you're asking an agent to do," he said. While routine data updates can be approved automatically, more consequential ones -- such as significant changes to sensitive master data or a supplier contract -- should be subject to human oversight, according to Jain.

Starting with a clear business case and a well-defined scope for an agentic AI deployment gives governance and software engineering teams a sound basis for proceeding with the rollout while building in sufficient controls, he added.

Earlier this year, OpenAI agents using the company's frontier AI models to run a cybersecurity training task escaped from an isolated testing environment and breached systems at Hugging Face, which operates an AI collaboration platform. It was just the first of various rogue-agent incidents that have now been disclosed by both OpenAI and AI rival Anthropic.

Asked during a Q&A segment about how to stop agents from bypassing restrictions to accomplish their tasks, Jain said organizations must be "very cognizant" of what's needed to prevent AI from taking control of systems. In response to a previous question about avoiding AI hallucinations, he pointed to an approach that might also help keep agents in line, noting that he breaks down agent workflows into smaller tasks with narrow scopes for testing, "so you can identify where things fail and what needs to be addressed."

Changes needed in data governance processes

Kelle O'Neal, founder and CEO of consulting firm First San Francisco Partners, said during a panel discussion that AI is now the key driver of data governance initiatives. That requires an increased focus on two areas traditional governance programs often ignore: metadata and unstructured data. Effective metadata management is critical for accurate AI outputs, O'Neal said, and governance teams can no longer treat unstructured data -- which agents increasingly rely on -- as "a second-class citizen."

Anant Somvanshi, a senior specialist in the Data & AI Defense Office at financial services firm Vanguard, said data privacy and security protections also need to change.

AI agents are not geniuses. … They are, however, relentless at doing what AI agents do. If that doesn't cause harm, great. If it does, uh-oh.

In addition to monitoring the use of sensitive data by humans, privacy now also involves controlling data use by AI models and whether the models can infer new information while doing so, Somvanshi said during the panel discussion. He added that security requirements have expanded from protecting data against breaches to defending AI systems against risks such as data poisoning and prompt injection attacks.

Johnson recommended combining strong data governance and identity management controls with a "zero-trust on steroids" security policy. Ideally, that would include the ability to shut down an agent "the millisecond it might be considered dangerous," she said -- but an AI kill switch of that sort isn't possible with existing technology.

For now, Johnson said governance and security leaders should grill technology vendors about their plans for better protecting AI systems against attacks. In particular, she cited patches for the vulnerabilities being identified in software products by Anthropic's Claude Mythos model as part of the AI vendor's Project Glasswing initiative.

Somvanshi said the fast pace of agentic AI development in business units often leaves data and AI governance teams "chasing them and trying to minimize the gap" between deployments and controls. What would be best, he added, is for organizations to move from centralized and federated governance models to one in which governance processes are integrated directly into development and deployment pipelines. That approach is known variously as embedded or shift-left governance.

Risk-based review processes for AI projects

While new strategies and practices are needed, Somvanshi said organizations can also accelerate AI governance implementations by reusing existing governance structures and fine-tuning them as necessary. In addition, not every AI use case requires the same level of review by governance teams before deployment, he said, suggesting a fast-path review process for low-risk applications and a more rigorous one for higher-risk projects.

ExxonMobil has adopted a similar approach in its enterprise AI governance program, according to Martin Obiozor, a data management solution architect at the oil and gas company. Obiozor said during a presentation that security controls advisors review proposed AI use cases with potential business value of $500 million or more, but not the ones below that threshold.

The governance team is also looking to accelerate its reviews of low-risk projects, he said. Currently, they go through the same process as higher-risk ones, starting with an intake survey with more than 150 questions. That's slowing AI development, which could become a problem if people look to go around the review process in the name of expediency, Obiozor noted.

But the average time it takes to complete project reviews has decreased from more than six months to about 90 days, he said. The governance team's ultimate goal is to reduce it to one to two weeks while also reducing the number of questions in their intake survey by as much as 70% -- all part of an effort, Obiozor said, to improve the AI governance process and make it more efficient.

Craig Stedman is a senior reporter on TechTarget's Data Technologies news team, covering analytics, data management and data backup topics. He has more than 40 years of experience as an enterprise IT writer and editor.