Organizations will need to rethink how they govern AI as autonomous agents increasingly interact across enterprise systems.
In a report released on Tuesday, Paris-based consulting firm Capgemini identified growing AI autonomy as a defining shift for enterprise technology, with implications across cybersecurity, R&D and commerce.
Overseeing interactions among agents will be one of the biggest challenges. The firm estimates that an enterprise running 1,000 agents alongside 200 agentic systems could face almost 720,000 potential interactions, any of which could produce unintended outcomes if not properly governed.
New controls
In response, Capgemini recommends controls at these levels: safeguards around individual agents, oversight of each agentic system and an enterprise-wide control plane. This would provide visibility into deployed agents, their permissions and interactions and resource consumption, alongside mechanisms to restrict or stop their activity.
Mark Beccue, an analyst at Omdia, a division of Informa TechTarget, said managing AI will be a central consideration for businesses implementing the technology.
“Unquestionably, we are entering the era of managing AI systems,” he said. “Businesses should have AI observability and AI management platforms to govern AI applications and agents.”
The biggest challenge for organizations scaling agentic AI over the next year will be “minimizing risk and maximizing outcomes” and finding a control plane to manage their AI systems, Beccue said.
“There’s a lot we don’t know yet about how autonomous agents interact with each other, but one we will be watching closely is how agents behave across disparate systems and how permissions and data privacy and protections will work when they disperse widely,” he said. “There are already some solutions that are on the market today that are designed to observe, monitor and manage agents.”
Confining failures
In the report, Capgemini also predicts a shift in AI safety practices, from relying on explainability to containing failures.
As models grow more sophisticated, organizations may not be able to inspect every step of their decision-making. Instead, businesses will need to watch for warning signs such as unauthorized tool use, unexpected changes in agent behavior, rising costs or slower response times, and have mechanisms to intervene.
Cybersecurity is another area in which Capgemini expects AI autonomy to have major consequences. AI agents are increasingly able to identify software vulnerabilities and develop exploits, potentially enabling attacks with little or no human involvement.
Scarlett Evans is a freelance writer with a focus on robotics and emerging technologies. Previously, she was assistant editor at IoT World Today, where she specialized in robotics and smart city technologies. She also has a background in the mining and resources sector, with experience at Mine Australia, Mine Technology and Power Technology.