Part of:Data observability boosts data pipeline performance
7 expert recommended data observability tools
Commercial data observability tools can offer organizations pre-built components and plenty of vendor support for data use cases including monitoring, security and decision-making.
Analysts and end users have sought data observability for years, but recent shifts have changed how business processes use these tools. This leaves organizations with plenty to consider when selecting a tool and deciding if commercial investment is worth it.
One major consideration is the move from watching infrastructure to watching the data itself. Another is the arrival of GenAI and agentic workloads for consuming and transforming data products.
Data observability tools have traditionally focused on capturing and analyzing log data to improve application performance monitoring and security. Data observability turns the focus back on the data to improve data quality, tune data infrastructure, and identify problems in data engineering pipelines and processes.
"Data analysts and business users are the primary consumers of this data," said Steven Zhang, director of engineering at Maven Clinic. "But it's becoming increasingly common that data engineers, who produce this data alongside product engineers, are also struggling with it."
This calls into question the trustworthiness of the data in terms of accuracy, reliability and freshness. This is where data observability tools come into play.
A good data observability tool captures these problems and presents them in a clean structure. It helps consumers understand conceptually where the data went wrong and helps engineers identify the root causes.
Why choose a commercial tool?
There are many open source and commercial tools available for organizations implementing data observability workflows. Commercial tools can fast-track this process with prebuilt components for common workflows and included vendor support. They also offer increased support for enterprise use cases like data quality monitoring, security and improved decision-making.
"A modern data infrastructure is often a combination of best-in-class but disjointed set of software environments that requires to be monitored and managed in a unified manner," said Sumit Misra, vice president and business leader for data and analytics partnerships at OwlSure, an IT consultancy and services provider.
For example, when a data job fails in one environment, another seemingly unrelated data environment must detect and react to the job's failure. Observable, responsive and self-treating data flows are becoming essential for businesses.
Commercial data observability tools can help organizations accelerate their time to deliver value from data quality initiatives, particularly when they are small or employ more business talent than IT talent, Misra said.
What to look for in a data observability tool
Enterprises often end up deploying more tools than they need or incorporating tools that are not specific or relevant to their business cases.
"Investments in commercial data observability tools and initiatives need to be made from the perspective of the overall business, internal users and customers," said Alisha Mittal, a vice president in IT services at Everest Group.
More tools do not always mean higher visibility. In fact, at times, these tools increase the system's complexity. Enterprises should strategically invest in observability tools by examining their current architecture, IT operations landscape and the skill development training and hiring required to handle the tools.
Various data quality and security functions are conventionally performed by an organization's data teams. However, the value of data observability tools lies in how these activities fit into the end-to-end data operations workflow and the level of context they provide on data issues.
Enterprises should consider how different data observability functions align with the following data quality management processes, Mittal said:
Monitoring offers a functional perspective of enterprise data systems or pipelines.
Alerting produces alerts/notifications both for expected events and anomalies.
Tracking provides the ability to set and track specific data-related events.
Logging keeps a record of events in a consistent way to facilitate quicker resolution.
Analysis involves an issue detection mechanism that provides insight on data pipelines and logs.
Agentic data observability
Recent innovations in GenAI and agents are imposing new requirements on data quality and oversight. One big challenge is that they may lack the context to assess when numbers start to look absurd, requiring additional measures to assess the quality of numbers that go into subsequent processes that a human may never see directly.
A perfectly healthy data pipeline can now produce a completely wrong business outcome.
Yulia Plugatyreva, senior SOX and IT auditor, Chime
"We used to observe data. Now we need to observe decisions," said Yulia Plugatyreva, a senior SOX and IT auditor at Chime. "A perfectly healthy data pipeline can now produce a completely wrong business outcome."
Ewelina Hayat, a consultant at technology research and advisory firm ISG, provided one example: A company might build an AI assistant over its policies and standard operating procedures, so call center employees can ask questions in natural language to deliver answers with the source and effective date attached. But when a policy changes, the document must be updated to drive updates across the entire supporting data pipeline for the new workflows, including ingestion jobs, vector embeddings enrichment, and vector database updates. But gaps in the process could mean everything appears to run correctly and the vector index job reports success.
"But if the new document hasn't actually made it into the retrieval index, the agent can still receive yesterday's policy," Hayat said. "There is no traditional pipeline failure. Everything is green. The AI simply gives the wrong answer with confidence."
Teams need visibility across a broader data processing estate to assess new processes that may confound agentic processes and the humans trying to make sense of their results. Vendors are beginning to ship agents as part of their platforms to diagnose and investigate problems, but caution is warranted, since automated remediation of one problem can introduce others.
Plugatyreva's rule is to automate investigation aggressively, but to automate the action itself only when it is reversible, bounded and routine. "In fintech, 'the agent was 95% confident' is not a great explanation for why a customer-facing or financial process changed," she said.
Top commercial data observability tools
Here are some of the top commercial data observability tools based on interviews with experts and users. These options are good at addressing enterprise considerations around investment, implementation and viability, Mittal said.
They also include a well-defined, value-based approach that aligns with business goals like operational efficiency or cost savings, and have prebuilt tool stacks to help enterprises realize immediate value. These tools focus on data observability specifically and are seeing enterprise adoption.
Acceldata
Acceldata has various tools to provide data observability for cloud, hybrid, and on-premises environments. Top capabilities include data pipeline monitoring, end-to-end data reliability and quality, multilayer data observability, extensive cloud capabilities, compute and cost optimization, and rapid setup. Coverage now extends beyond database tables to unstructured and streaming data.
The company has expanded into agentic data management. The data observability capabilities are now one module in a wider platform that also covers pipeline engineering, cataloging, governance and a runtime for AI agents. It has also added a separate line of AI observability capabilities that trace prompts, models, retrieval steps and tool calls. Its data quality, pipeline, catalog, reconciliation and incident management functions are packaged as agents that support human in the loop workflows.
Acceldata is good for acquiring thorough, cross-functional visibility into complicated, frequently interrelated data systems, Mittal said. This makes it the preferred observability tool in the payments and financial sector. It also excels in combining signals from many tasks and layers on a single pane of glass. This allows different teams to collaborate more efficiently.
One caveat is these tools might not be preferable for enterprises using many different external monitoring tools, Mittal said.
Anomalo
Anomalo uses a machine learning approach to automate data observability processes to learn normal behavior and identify anomalies. This reduces the burdens on teams to write rules manually for every check. It’s aimed at organizations with large data estates and small teams that may struggle with monitoring large data pipelines in production. These capabilities extend to monitoring unstructured data with tools for evaluating document collections and scoring them for ingestion into language models for flagging problems such as missing metadata, unreadable or corrupted formats, duplication, personally identifiable information, and proprietary content. This can help build retrieval-augmented generation pipelines for internal documents.
Recent progress has included agentic tools that automate detection, triage, and root cause analysis. This can help investigate and recommend mitigations or fixes for data engineering teams. These tools can run natively across major cloud data warehouses and lakehouses. Costs are tied to the volume of data monitored. Some FinOps discipline should be considered in scoping datasets that warrant continuous monitoring before rolling it out broadly.
Elementary
Elementary has focused on bringing data observability into data pipelines rather than adjacent to them. Its open source package can be installed into existing or new dbt projects to collect metadata, run results, and test outcomes during normal runs. These tools also support various tests for volume, freshness, distribution, and schema changes. One advantage is that these can run as native dbt tests, which simplifies workflows for versioned anomaly detection using existing infrastructure-as-code processes used by data teams.
The open source tools can run on their own for generating reports and sending alerts via Slack and Microsoft Teams. A commercial cloud service adds column-level lineage tracking, catalog management, automated monitoring, and agent-driven anomaly investigation. It’s a good fit for teams that have standardized their data transformation pipelines on dbt.
IBM Data Observability by Databand
IBM Data Observability by Databand, which is built on technology acquired with its purchase of Databand in 2022, is a data observability platform to help teams detect and resolve data issues. One top feature is support for proactive capabilities to help detect data and resolve data incidents earlier in the development cycle. It includes tools for collecting metadata, profiling behavior, detecting and alerting on data incidents, and triaging data quality issues. It can be a good choice for companies with an extensive IBM infrastructure.
In addition, IBM now offers data observability as a service within IBM watsonx.data integration. This is part of a broader trend within the industry as data observability moves from being a standalone product to a capability built into some of the larger data engineering platforms. IBM watsonx.data integration provides a unified control plane for batch, streaming, replication and unstructured data pipelines. The data observability features monitors the health of integration flows alongside the tools that build them. AI-assisted troubleshooting capabilities help teams investigate failures and data quality problems.
Monte Carlo
Monte Carlo provides a comprehensive data observability capability. It's an end-to-end platform focusing on fixing faulty data pipelines, Mittal said. It helps engineers ensure dependability and troubleshoot issues before they cause an outage. Top features include data catalogs, automated alerting and observability on several criteria. It also supports a fully automated setup.
The company has since extended well past pipeline monitoring. It added monitoring for unstructured data such as documents, chat logs and images, without requiring users to write SQL. It has also extended data observability to AI agents themselves to assess the context an agent retrieves, its behavior and tool calls, its performance and cost, and the quality of its outputs.
In parallel with these moves, the company has begun characterizing itself as an agent trust platform, which represents an effort to move beyond the traditional data observability framing. It has also built integrations with the major cloud data platforms' agent frameworks.
Precisely
Precisely provides a data catalog and data integrity suite. Its data observability capabilities help teams identify impacts from adverse data integrity issues. It focuses on ease of use, intelligent analysis that minimizes alert fatigue, and interoperability with modern tech stacks and data infrastructure. It extensively uses AI and ML capabilities to identify issues and root causes. It links data observability capabilities into a data catalog, helping teams find and integrate data into new workflows. This improves the quality of data in workflows involving enrichment with business, location or consumer data.
Precisely has moved to package AI capabilities as assistants and agents rather than as background machine learning. The suite includes a conversational assistant. Various specialized agents support data quality, data integration, enrichment and location intelligence workflows. These can help with configuring replication pipelines, mapping schemas and proposing quality rules. Audit trails and configuration controls support GRC requirements for regulated industries. It has also opened the suite to external AI systems, exposing its APIs through a hosted Model Context Protocol server to allow agents and assistants built elsewhere to discover and use its data quality, enrichment and location capabilities. This combination is framed as supporting agentic-ready data.
Soda Data Quality
Soda is an AI-powered data observability platform. It also includes extensive collaboration capabilities to help data owners, data engineers and data analytics teams work through issues.
This platform targets sophisticated data consumers. Enterprises can rapidly examine enterprise data right away, define rules to test and validate data and respond programmatically anytime a test fails.
For instance, enterprises can immediately halt data operations and quarantine data. Checks are written in a declarative language designed for human readability that can be stored in version control with other code artifacts. The core engine is free to self-host and released under a source-available license with some restrictions compared to more permissive open source licenses. The company is headquartered in Europe and offers a self-hosted runner that may appeal to European organizations concerned about data sovereignty requirements.
George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.
