Monte Carlo's Agent Observability targets reliability of AI
The vendor's new feature enables users to define acceptable AI outputs and trace outputs that don't meet organizational standards back to source data so they can be addressed.
Monte Carlo on Tuesday launched Agent Observability, a new capability that combines data and AI observability to help customers develop dependable agents and other AI applications.
Previously, the vendor's capabilities helped users oversee data management processes such as ingestion and transformation but did not address AI processes. By unifying the two, Agent Observability -- which is not itself an agentic AI-powered feature -- enables customers to not only monitor data to ensure its quality when informing AI and analytics tools but also monitor retrieval from knowledge sources and output quality that are key to AI development.
Given that AI models and applications need high-quality data to be accurate and effective, Monte Carlo is "on the right trajectory" with its launch of Agent Observability, according to Stewart Bond, an analyst at IDC.
"There is no AI without data, and the combination of data and AI is something that needs to be monitored and continuously evaluated," he said. "Mechanisms will need to be put in place to take control of assistants, advisors and agents to keep humans in the loop. … This is where end-to-end visibility is important because it provides a holistic view of the inputs and outputs together."
Based in San Francisco, Monte Carlo is a data observability provider of tools that enable customers to oversee their data estates and to ensure data quality.
Unified data & AI observability
With generative AI (GenAI) capable of making workers better informed and more efficient, many enterprises have been increasing their investments in AI development since OpenAI's November 2022 launch of ChatGPT.
However, despite steadily rising interest in AI development, an overwhelming majority of AI projects fail to make it into production.
According to a PwC survey in May of more than 300 senior executives, nearly half plan to increase their AI-related budgets by 10% to 25% over the next 12 months with another quarter of respondents increasing their budgets by more than 25%. Barely one-third, however, report broad adoption of agents throughout their organization.
Meanwhile, overall, it is estimated that over 80% of AI projects never make it into production.
While not the only culprits, two reasons that adoption trails investment are the interrelated issues of poor data quality and lack of trust in agents. Monte Carlo's Agent Observability unifies data observability to address data quality and AI observability to engender trust in AI applications.
As a result, like Bond, Kevin Petrie, an analyst at BARC U.S., noted the importance of tools like Agent Observability.
"Monte Carlo users, like all organizations, must extend their data governance programs to address new risks as they adopt agentic AI," he said. "They can use one platform to govern both the quality of model inputs and the suitability of model and agent outputs."
Agent Observability uses LLM-as-judge monitors to evaluate AI outputs. In addition, Monte Carlo customers can customize prompts to define acceptable outputs. In both instances, AI outputs are mapped back to the source data, enabling users to address problems when alerted.
"Many faulty outputs derive from faulty inputs," Petrie said. "By tracing the full lineage of such issues, Monte Carlo users can remediate future risks."
While statistics such as those cited by PwC demonstrate how difficult it is for organizations to build AI capabilities, customer feedback fueled Monte Carlo's development of Agent Observability, according to Barr Moses, the vendor's co-founder and CEO.
"When Monte Carlo first launched, the main focus of data teams was powering internal BI and customer-facing data products. But with the rapid rise of AI products and agents, customer needs shifted," she said.
Monte Carlo heard customers mention challenges around using AI to increase productivity, ensure data is trusted for AI and successfully ship AI products, Moses continued.
"Monte Carlo already addressed the first two. ... The natural next step was to help customers get AI into reliable production," she said.
Beyond its significance for Monte Carlo customers, the vendor's unification of observability across data management and AI capabilities in a single tool could help Monte Carlo differentiate itself from some of its competition, according to Bond.
Monte Carlo's competitors include Acceldata, Anomalo, Bigeye, Datafold, Metaplane and Soda Data. Among those, only Acceldata offers something like Agent Observability, providing data and AI observability within its Agentic Data Management Platform, although not within a single feature.
"Monte Carlo may be one of the first to package the capabilities as a product offering," Bond said.
However, whether Agent Observability is truly an effective judge of AI outputs remains to be seen, he continued. In addition, whether Agent Observability can adequately observe agents is also to be determined, he said.
Some vendors engage in "agent washing," or exaggerating AI capabilities to include agentic AI when, in practice, they lack such capabilities.
"Monte Carlo mentions end-to-end visibility across data and AI, but AI is not synonymous with agents," Bond said. "There is a lot of hype around agents, but most of what organizations consider agents are still assistants or advisors."
Looking ahead
With Agent Observability now available, Monte Carlo plans to continue developing capabilities aimed at better enabling customers to successfully develop AI tools, according to Moses.
"Monte Carlo plans to … focus on more advanced support for agentic workflows and even greater ease of use in deploying monitoring and troubleshooting capabilities," she said.
Petrie, meanwhile, suggested that Monte Carlo broaden its ecosystem for AI observability by partnering with AI and machine learning platforms.
"This will help its data engineering users collaborate more easily with the data scientists, ML engineers and developers that manage agentic AI projects," he said.
Bond similarly advised Monte Carlo to expand its capabilities through partnerships. Specifically, the vendor would be wise to address data integrity and protection by partnering with data intelligence and AI governance software vendors, according to Bond.
"Not only is it important to observe the quality of inputs and outputs, but also assure privacy, security and toxicity are being monitored for compliance with regional regulations and corporate policies," he said.
Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than 25 years of experience. He covers analytics and data management.
