Copyright TechTarget - All rights reserved https://cyber.law.harvard.edu/rss/rss.html Techtarget Feed Generator en Wed, 19 Aug 2026 17:17:11 GMT https://www.techtarget.com/data-technologies editor@techtarget.com <p>Boards are demanding clearer ROI from data investments as AI spending rises, but many chief data officers still lack the metrics to prove their value in financial terms.</p> <p>Capital One is often recognized as one of the first major companies to appoint a CDO in 2002. Financial services firms adopted the role to address mounting concerns for data governance and regulatory compliance. But the job has moved well beyond defensive oversight.</p> <p>Today, CDOs are expected to help turn enterprise data into business value, especially as organizations move their AI pilots into production. An IBM <a target="_blank" href="https://www.ibm.com/downloads/documents/us-en/1443d5f8d4cf5197" rel="noopener">study</a> from 2025 found that 92% of CDOs say they must focus on business outcomes to succeed, yet only 29% have clear measures to determine the value of those outcomes.</p> <p>A four-part methodology can help CDOs demonstrate the value of their work in terms that executives understand. The goal is to move from traditional technical metrics and show the financial value the data function brings to the organization.</p> <section class="section main-article-chapter" data-menu-title="1. Stop reporting technical activity"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Stop reporting technical activity</h2> <p>Even in technology companies, executive boards care more about income statements than architecture diagrams. To a CDO, technical and architectural measures, such as server uptime, query latency and <a href="https://www.techtarget.com/it-infrastructure/tip/6-cloud-migration-challenges-to-prepare-for-and-overcome">cloud migration progress</a>, are important to data teams, but they track effort, not results. Every technical measure should map to a business outcome that the board can act on.</p> <p><iframe title="Technical output translated to business outcomes" aria-label="Table" id="datawrapper-chart-T9vvr" src="https://datawrapper.dwcdn.net/T9vvr/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="416" data-external="1"></iframe></p> <p> <script type="text/javascript">(function(){function e(){window.addEventListener(`message`,function(e){if(e.data[`datawrapper-height`]!==void 0){var t=document.querySelectorAll(`iframe`);for(var n in e.data[`datawrapper-height`])for(var r=0,i;i=t[r];r++)if(i.contentWindow===e.source){var a=e.data[`datawrapper-height`][n]+`px`;i.style.height=a}}})}e()})();</script> </p> <p>To avoid reporting only in technical terms, <a href="https://www.techtarget.com/searchapparchitecture/tip/The-key-to-aligning-technology-initiatives-with-business-goals">connect every initiative to a stated business goal</a>. For example, if the CEO names customer retention as the year's priority, the data program's job is to predict and reduce churn, not migrate servers.</p> <p>Some CDOs enforce a business sponsor rule: no project proceeds unless an executive outside the data function -- whose own performance targets depend on the result -- commits to it. Executive sponsorship secures adoption, which in turn creates a clearer path to measurable value.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Report against baselines"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Report against baselines</h2> <p>A CDO can't claim reductions in data processes without first measuring the baseline. Before any initiative starts, record the current state. Common scenarios include a <a href="https://www.techtarget.com/searchbusinessanalytics/opinion/Will-AI-replace-data-analysts-A-year-and-a-half-later">data analyst's hours</a> spent locating, verifying and correcting data, or the number of data errors in executive reports that require fixes.</p> <p>Baselines will matter even more as companies deploy AI agents that act on data directly, for three reasons:</p> <ol class="default-list"> <li><b>Comparison.</b> Without baseline measurements, organizations can't determine if the agent produces better results.</li> <li><b>Scope.</b> AI agents don't just accelerate existing processes; they introduce new activities that prior measurements didn't cover. For example, how many times <a href="https://www.techtarget.com/searchenterpriseai/feature/Humans-and-AI-The-role-of-people-in-the-new-AI-world">a human must intervene to correct an agent</a> or the compute cost of a completed task. Those indicators need a recorded starting point because they can change over time as systems, models and workflows evolve.</li> <li><b>Speed and scale. </b>AI agents can magnify the impact of errors as they act across more data, systems and decisions.</li> </ol> </section> <section class="section main-article-chapter" data-menu-title="3. Track progress and settle attribution in advance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Track progress and settle attribution in advance</h2> <p>Reporting against baselines works best on a fixed cycle, using red, amber or green status for each funded initiative. Attributing changes in metrics is more difficult. If the data team gives marketing a better, unified customer record and sales conversion rates improve, who gets the credit? The VP of sales might win the argument -- they are, after all, good at selling!</p> <p>Negotiate how improvements will be attributed before the project starts. Outline what teams should expect from planned improvements, including new contributions to operations, and document those agreements. To gain more insights, run controlled A/B tests to isolate factors so the claim withstands scrutiny.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Price the work across four dimensions"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Price the work across four dimensions</h2> <p>CDOs can quantify data ROI across four dimensions: operational efficiency, revenue enablement, risk mitigation and <a href="https://www.techtarget.com/searchdatamanagement/feature/Reconsider-the-AI-readiness-gap-in-data-and-analytics">AI readiness</a>. Each dimension requires a distinct pricing method.</p> <p><iframe title="Quantifying ROI across four dimensions" aria-label="Table" id="datawrapper-chart-niudV" src="https://datawrapper.dwcdn.net/niudV/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="585" data-external="1"></iframe></p> <p> <script type="text/javascript">(function(){function e(){window.addEventListener(`message`,function(e){if(e.data[`datawrapper-height`]!==void 0){var t=document.querySelectorAll(`iframe`);for(var n in e.data[`datawrapper-height`])for(var r=0,i;i=t[r];r++)if(i.contentWindow===e.source){var a=e.data[`datawrapper-height`][n]+`px`;i.style.height=a}}})}e()})();</script> </p> </section> <section class="section main-article-chapter" data-menu-title="5. Tailor communications to stakeholder concerns"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Tailor communications to stakeholder concerns</h2> <p>Every data investment affects executives differently. CDOs can <a href="https://www.techtarget.com/it-strategy/tip/How-to-present-to-the-board-of-directors-15-tips-for-a-successful-presentation">tune reporting carefully</a> without overstating results for specific audiences.</p> <p><iframe title="Examples of tailoring reports to your audience" aria-label="Table" id="datawrapper-chart-SsHRK" src="https://datawrapper.dwcdn.net/SsHRK/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="452" data-external="1"></iframe></p> <p> <script type="text/javascript">(function(){function e(){window.addEventListener(`message`,function(e){if(e.data[`datawrapper-height`]!==void 0){var t=document.querySelectorAll(`iframe`);for(var n in e.data[`datawrapper-height`])for(var r=0,i;i=t[r];r++)if(i.contentWindow===e.source){var a=e.data[`datawrapper-height`][n]+`px`;i.style.height=a}}})}e()})();</script> </p> <p><em>Donald Farmer is a data strategist with more than 30 years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> CDOs can defend the spend on data programs by tying outcomes to saved time, lower risk, revenue gains and AI projects that reach production. https://cdn.ttgtmedia.com/rms/onlineimages/collab_a102381789.jpg https://www.techtarget.com/data-technologies/tip/How-CDOs-can-show-the-business-impact-of-data Wed, 19 Aug 2026 15:40:00 GMT <p>Data observability specialist Bigeye is lasering in on the cost of AI.</p> <p>Many enterprises are <a href="https://www.techtarget.com/enterprise-software/feature/AI-feature-spend-is-the-new-software-cost-control-problem">paying more attention to their spending</a> on AI development and management as they move AI initiatives out of the experimentation phase, when the primary focus is on ensuring that agents and other AI tools work in real-world situations, and into production environments.</p> <p>In response, Bigeye on Wednesday launched Cost Anomaly Detection within its Agent Trust Hub to automatically alert customers when agents and other data and AI consumers start spending more or less than their expected levels. Previously, a human would have had to manually monitor or audit an agent or user's behavior to catch sudden <a href="https://www.techtarget.com/it-strategy/feature/The-Meta-warning-When-AI-spending-becomes-a-liability">spending surges</a> or declines.</p> <p>Stewart Bond, an analyst at IDC, noted that his firm's research shows that two-thirds of the organizations it surveyed for a report released in July spent more on agentic AI than they budgeted.</p> <p>"Every dollar of wasted AI agent spend is a governance failure," Bond told TechTarget. "You cannot manage what you don't measure, and many organizations still aren't measuring agent spend in a way that catches problems as they happen, rather than a month later on an invoice."</p> <p>What is particularly valuable about Bigeye's new feature is that it surfaces not only AI spending increases but also decreases, he continued.</p> <p>"Catching an agent whose cost has quietly dropped to zero … is a different and often overlooked failure mode," Bond said. "Baselining against an entity's own history, with no budget model to configure upfront, also lowers the bar to actually turning this on, which matters given how many organizations already own cost dashboards they aren't fully using."</p> <p>Based in San Francisco, Bigeye is a <a href="https://www.techtarget.com/data-technologies/tip/The-5-pillars-of-data-observability">data observability</a> vendor that competes with fellow specialists such as Monte Carlo, along with broader-based data management providers including IBM and Informatica. In June, Bigeye launched Agent Trust Hub, a centralized location where users can observe agents, their activity and the <a href="https://www.techtarget.com/data-technologies/news/366646252/Context-is-king-as-agents-evolve-but-problems-persist">underlying data that informs them</a> to understand whether the AI tools can be trusted.</p> <section class="section main-article-chapter" data-menu-title="Addressing AI spending"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Addressing AI spending</h2> <p>The high cost of AI development and deployment is becoming a focal point for many enterprises. If spending spirals out of control, as it did for companies such as <a target="_blank" href="https://finance.yahoo.com/technology/ai/articles/canva-rare-startup-grew-fast-201427317.html" rel="noopener">Canva</a> and <a target="_blank" href="https://www.tradingview.com/news/reuters.com,2026:newsml_L4N4422AW:0-figma-s-upbeat-outlook-fails-to-stem-margin-worries-as-ai-costs-mount-shares-slump/" rel="noopener">Figma</a>, there can be significant consequences.</p> <p>Developing an agent, which involves model routing,&nbsp;context management and processing large amounts of data, can cost well over $100,000, according to IT consulting firm Triple Minds. Once in production, orchestration and continued calls to models and other tools add further expenses.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Every dollar of wasted AI agent spend is a governance failure. You cannot manage what you don't measure, and many organizations still aren't measuring agent spend in a way that catches problems as they happen, rather than a month later on an invoice. </figure> <figcaption> <strong>Stewart Bond</strong>Analyst, IDC </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>According to <a href="https://www.ey.com/en_us/newsroom/2026/07/ey-survey-c-suites-pivot-from-ai-adoption-to-unlocking-value-as-escalating-token-costs-trigger-fiscal-scrutiny">the fifth wave of EY's US AI Pulse Survey</a>, released in July, 82% of the surveyed senior leaders whose organizations are investing in AI report being concerned about their use of AI tokens and the cost of doing so.</p> <p>As a result, data and AI providers including AWS, Databricks, Microsoft and <a href="https://www.techtarget.com/data-technologies/news/366649521/Snowflake-targets-cost-of-AI-with-dynamic-model-routing">Snowflake</a> have all introduced features in recent months designed to help customers control their spending on AI.</p> <p>Bigeye is doing the same with Cost Anomaly Detection, with customer feedback motivating the vendor to develop the new feature, according to Bigeye CEO Eleanor Treharne-Jones.</p> <p>"As enterprises move AI agents from pilot into production, cost becomes hard to attribute -- a workflow's token usage can spike, or quietly drop to near-zero, and teams often don't notice until the bill lands or the agent stops delivering value," she told TechTarget. "We heard this directly and repeatedly from our own customers … and saw it validated more broadly across the market."</p> <p>Cost Anomaly Detection not only enables users to understand what they spent on AI, but when spending changes and why it does so. Each AI agent interaction that Bigeye observes has an estimated cost. When that cost changes beyond its expected range, Cost Anomaly Detection sends an alert that includes details about the AI actions that resulted in the disparity.</p> <p>In particular, Cost Anomaly Detection can discover when an agent slips into a retry <a href="https://www.techtarget.com/whatis/definition/loop">loop</a> or an unusually interactive conversation pattern with other applications, and costs suddenly increase. In addition, it can catch an agent that stops conversing with other tools because a credential expired or <a href="https://www.computerweekly.com/news/366643493/Avoid-expensive-AI-agents-with-these-five-design-imperatives">an integration failed</a>.</p> <p>"Cost Anomaly Detection is a real addition," Mike Leone, an analyst at Moor Insights &amp; Strategy, told TechTarget. "Most cost tools will catch an agent that suddenly triples its spend, and Bigeye does that. Catching an agent whose cost drops to zero is harder, and that usually means the agent stopped working. A falling bill looks like savings until somebody notices the work stopped and was never completed."'</p> <p>Monitoring for AI spending anomalies is not, however, a differentiator for Bigeye, he continued. But there is differentiation in the level of detail.</p> <p>"Most AI observability tools already track token spend," Leone said. "Where Bigeye does something less common is the attribution. It ties agent spend to the specific tables an agent queried, using the access records it already collects to enforce policy. It also works against older systems that predate the cloud warehouses."</p> <p>Bond likewise noted that, while the addition is valuable for Bigeye customers, <a href="https://www.techtarget.com/data-technologies/news/366630477/Monte-Carlos-Agent-Observability-targets-reliability-of-AI">Monte Carlo</a> also provides features that monitor data, code, AI models and infrastructure to analyze costs. Similarly, <a href="https://www.techtarget.com/data-technologies/news/366623394/Acceldata-launches-agentic-AI-powered-anomaly-detection">Acceldata</a> alerts users to cost overruns and performance degradation.</p> <p>However, Bigeye is distinguishing itself from competing vendors by integrating anomaly-based cost detection with existing data lineage and governance capabilities, Bond continued. With Cost Anomaly Detection, a change in the cost of an agent can be seen in conjunction with what data the agent accessed and whether that <a href="https://www.techtarget.com/data-technologies/opinion/Generative-AI-shines-spotlight-on-data-governance-and-trust">data can be trusted</a>.</p> <p>"The market's real gap isn't visibility anymore," he said. "It's turning visibility into enforceable outcomes, such as budget guardrails and automated rightsizing. That is likely where the real competitive separation will happen next, and it isn't fully solved by Bigeye or anyone else yet."</p> </section> <section class="section main-article-chapter" data-menu-title="Next steps"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Next steps</h2> <p>After launching the Agent Trust Hub and adding Cost Anomaly Detection over the past few months, Bigeye's focus for the remainder of 2026 is to add more capabilities that enable customers to understand and <a href="https://www.techtarget.com/searchenterpriseai/post/How-CIOs-should-architect-trust-in-AI-not-just-govern-it">trust their agentic AI tools</a>, according to Treharne-Jones.</p> <p>"We are focused on … deepening the trust and governance layer for agentic AI as more of these systems move into production, and expanding the platforms [to which] Agent Trust Hub connects to provide comprehensive visibility across the AI ecosystem," she said.</p> <p>Bond advised Bigeye to advance tools such as Cost Anomaly Detection from merely alerting users to anomalies to automatically resolving issues such agentic AI cost overruns.</p> <p>"Vendors need to shift from reporting features toward real budget guardrails and automated rightsizing, since the tooling for visibility already exists broadly in the market and isn't closing the value gap on its own," he said.</p> <p>In addition, adding capabilities that <a href="https://www.techtarget.com/ai/tip/The-best-AI-governance-tools-and-platforms-in-2026">trace every decision</a> and action made by AI tools would be valuable for Bigeye users, Bond continued.</p> <p>"IDC found that maintaining a complete auditable record of agent decisions is the fastest-rising pain point in agent operations, even as raw security concerns decline with maturity," he said. "A natural next step for Cost Anomaly Detection would be to tie anomalies not just to conversations but to a defensible audit trail of what the agent decided and why."</p> <p>Leone likewise suggested that Bigeye integrate policy enforcement with anomaly detection. He noted that the vendor already has a policy layer that sits between AI agents and data. Adding a feature that <a href="https://www.techtarget.com/ai/tip/Why-businesses-need-an-AI-agent-kill-switch">stops agents</a> from spending when they hit a certain level is a natural next step.</p> <p>"Flagging that an agent spent five times its normal amount yesterday helps, [but] cutting the agent off when it crosses twice its normal amount helps more," Leone said.</p> <p><i>Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.</i></p> </section> With agentic AI costing more than many organizations expected, the data observability specialist's latest alerts users when the expense of an agent suddenly changes. https://cdn.ttgtmedia.com/rms/onlineimages/money_g972609480.jpg https://www.techtarget.com/data-technologies/news/366649478/Data-observability-specialist-Bigeye-puts-focus-on-AI-spend Wed, 19 Aug 2026 14:35:00 GMT <p>For the better part of two decades, enterprise data strategy ran on a single assumption: more data meant better outcomes. AI has broken that assumption, and the organizations discovering it are learning that the bottleneck isn't volume, but context.</p> <p>The accumulation strategy was rational for the workloads it served. Enterprises consolidated data lakes, scaled warehouses and ingested everything available, because BI consumed structured data in predictable patterns. Having more data in one place made reporting faster and dashboards richer.</p> <p>That logic has started to break down. As enterprises move from BI-era workloads to agentic AI, the question is no longer how much data they have, but whether that data carries the context, definitions and governance required for an AI agent to use it correctly. The gap between what enterprises have built and what AI requires is widening, and closing it demands a fundamentally different kind of investment.</p> <section class="section main-article-chapter" data-menu-title="The workload that broke the old model"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The workload that broke the old model</h2> <p>Traditional BI workloads were structured, predictable and scoped. An analyst queried a warehouse, built a dashboard and refined a report. The data was organized for human consumption, and the questions it answered were framed by humans.</p> <p>AI workloads do not operate in the same way. Retrieval-based systems, including the <a href="https://www.techtarget.com/searchenterpriseai/tip/RAG-best-practices-for-enterprise-AI-teams">RAG architectures underpinning most enterprise agents</a>, assemble their context at query time, pulling from multiple sources, interpreting definitions on the fly and executing tasks without a human framing each query in advance. What the system retrieves in the moment determines the quality of what it produces. And unlike a human analyst who can recognize when a metric looks wrong, an agent will use what it is given.</p> <p>The difference has made context the highest-returning investment in the data stack. Felix van de Maele, CEO of Collibra, pointed to evidence from <a href="https://claude.com/blog/how-anthropic-enables-self-service-data-analytics-with-claude">Anthropic's own self-service analytics work</a>. Without business context, Anthropic's agents achieved roughly 20-25% accuracy on analytical tasks, van de Maele said. With governed business context layered in, accuracy climbed to 95%.</p> <p>The numbers suggest that improving the context fed to a model now produces a larger performance gain than improving the model itself or expanding the data it can access. For organizations that spent years investing in volume, the implication is uncomfortable. The constraint they optimized for is not what matters most for their current needs.</p> </section> <section class="section main-article-chapter" data-menu-title="How the gap became structural"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How the gap became structural</h2> <p>An important question to consider is why context and governance were neglected in the first place. Enterprises were not careless; the way organizations fund and deliver technology projects made reuse and <a href="https://www.techtarget.com/searchdatamanagement/feature/Why-enterprise-AI-depends-on-the-semantic-layer">shared semantic foundations</a> structurally difficult to build.</p> <p>Terry Dorsey, senior data architect at Denodo, spent much of her career in industry roles before moving to the vendor side. She described an environment in which every data initiative was scoped and funded as a standalone project, with resources tied to specific deliverables and no mechanism for cross-project reuse.</p> <p>"A lot of it has to do with how organizations structure: how they do funding, who gets funding and how do you use that funding," Dorsey said. Each new initiative rebuilt its own data foundations from scratch, even when an adjacent team had already done similar work. Over time, the pattern produced overlapping silos, duplicated logic and mounting technical debt that individual project budgets were never structured to address.</p> <p>The technology changed across eras, but the organizational pattern did not. Dorsey noted that at a recent industry conference, practitioners were already discussing <a href="https://www.techtarget.com/searchenterpriseai/tip/Controlling-AI-sprawl-A-practical-guide-for-the-C-suite">agent sprawl</a> as the latest iteration of a familiar cycle. Report sprawl became data sprawl, and data sprawl is becoming agent sprawl. The same project-scoped thinking that created redundant dashboards and disconnected data pipelines is now producing redundant agents with no shared governance layer.</p> <p>"Fundamentally, all we've done is move data from one place to another," Dorsey said. "You can get at it faster, but we haven't worked on how we actually deliver it to people. And we keep the same processes we've had for the past 20-plus years."</p> </section> <section class="section main-article-chapter" data-menu-title="The cost of the gap is visible"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The cost of the gap is visible</h2> <p>For years, the absence of shared semantic foundations was a latent inefficiency. AI has made it an active cost.</p> <p>Van de Maele described a "hallucination tax" that enterprises are paying <a href="https://www.techtarget.com/whatis/feature/How-companies-are-tackling-AI-hallucinations">when agents operate without governed context</a>. When AI <a href="https://www.techtarget.com/searchcio/news/366610973/Explainable-AI-systems-build-trust-mitigate-regulatory-risk">output cannot be trusted</a>, organizations default to human-in-the-loop verification, double-checking every result. That verification bottleneck erodes the productivity gains that justified the AI investment in the first place.</p> <p>The cost extends beyond labor. Agents consuming ungoverned data <a href="https://www.techtarget.com/it-strategy/feature/Tokenmaxxing-How-CIOs-can-extract-maximum-value-from-AI-tokens">burn tokens processing irrelevant or duplicative information</a>, driving up compute expenses without improving outcomes. Van de Maele said token efficiency has become a key metric alongside task completion because throwing more unstructured data at an agent makes the operation more expensive and degrades accuracy.</p> <p>The prototype-to-production bottleneck tells the same story from another angle. Van de Maele said getting agents from prototype to production "is turning out to be a lot harder than people expected," and attributed the difficulty to <a href="https://www.techtarget.com/it-strategy/feature/The-AI-agent-governance-gap-How-CIOs-can-gain-control">governance gaps</a> rather than technical limitations. The models work, but organizations are still building the context that makes them work reliably in production.</p> <p>Dorsey offered a practitioner's version of the same observation. Enterprises often believe that hiring skilled AI researchers would be sufficient to operationalize AI. Researchers could build models in weeks, but the surrounding work turned out to be the far larger component. Governance had to be established for what data an agent could expose. Business processes needed to change, and the organizations had to be brought along. The development was the smaller part of a much bigger effort that the organization had not anticipated.</p> </section> <section class="section main-article-chapter" data-menu-title="What the response looks like"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What the response looks like</h2> <p>No single playbook exists for closing the gap between what enterprises have built and what AI needs. Organizations are approaching the problem from different starting points, and the strategies reflect those differences.</p> <p>Some enterprises are beginning with discovery and classification, according to van de Maele. They inventory their data, tag and classify what they have and build a semantic structure before deploying AI against it. The emphasis is on understanding the estate before asking agents to operate within it.</p> <p>Others are working backward from a specific AI use case. Rather than cataloging everything, they identify a high-value workflow, build a curated knowledge base to support it and expand outward from there. This approach is narrower and faster to deliver results, though it risks recreating the project-scoped pattern Dorsey described if the knowledge base remains siloed within a single initiative.</p> <p>What both strategies share is a recognition that the next investment must go somewhere different. The volume era asked how much data an enterprise could bring together. What matters now is whether the data already in hand carries the context, definitions and governance to be trusted by the systems consuming it. That shift, not the failure of the old strategy, is what the next investment has to answer.</p> <p><em>Scott Thompson is the Site Editor for TechTarget's Data Technologies group, covering data management and business analytics topics for senior enterprise data leaders. He has edited data and analytics content for TechTarget since 2021.</em></p> </section> Volume-first data strategy served the BI era but falls short for agentic AI, where the constraint on readiness is context and governance rather than how much data exists. https://cdn.ttgtmedia.com/rms/onlineimages/folder-files11.jpg https://www.techtarget.com/data-technologies/feature/Why-more-data-will-not-deliver-AI-data-readiness Wed, 19 Aug 2026 09:32:00 GMT <p>Data pipelines have long been the foundation for BI, data analytics and business operations. They remain essential to providing trusted data to users, but AI changes what a pipeline architecture must deliver.</p> <p>Extract, transform and load (ETL) remains a fundamental data integration approach, but as <a href="https://www.techtarget.com/data-technologies/opinion/AIs-appetite-for-data-is-changing-data-requirements">AI workloads expand</a>, enterprises need a data pipeline architecture that also supports streaming and ELT, which reverses the load and transform steps. Stronger data validation, better observability, clearer lineage and stricter access controls are also required.</p> <p>The question for data teams is whether their current pipeline designs can deliver trusted data fast enough, with sufficient quality and transparency, for AI systems to act on it. If data arrives too late, schema changes require manual fixes, quality issues are caught only after delivery, or lineage and access controls don't extend to the AI retrieval layer, the pipeline likely needs a refresh.</p> <section class="section main-article-chapter" data-menu-title="What AI workloads require from data pipelines"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What AI workloads require from data pipelines</h2> <p>Five years ago, many enterprise data pipeline architectures were designed mainly around analytics refresh cycles. The target was usually a data warehouse or a BI dashboard. BI and analytics can tolerate some latency, and humans can make adjustments -- such as data analysts performing reviews and addressing issues -- to get the right results before they're used to influence business decisions.</p> <p>Autonomous AI agents have a different risk profile. AI agents make concurrent, multi-step data requests and can act on outputs with limited -- or no -- human review. As a result, a stale or malformed data input can propagate directly into AI model behavior or business decisions.</p> <p>In a March 2026 <a target="_blank" href="https://www.idc.com/resource-center/blog/ai-cant-run-on-stale-data-why-enterprises-are-rethinking-their-architecture/" rel="noopener">blog post</a>, IDC said it expects 80% of agentic AI use cases to require real-time, contextual and widely accessible data. The post also said the market research firm expects 40% of the top 2,000 public companies worldwide to adopt modern event streaming technology and prebuilt real-time data views to support AI agents by 2027.</p> <p>That shift changes the requirements for a data pipeline architecture:</p> <p><iframe title="The new standard for enterprise data flows" aria-label="Table" id="datawrapper-chart-0Rfna" src="https://datawrapper.dwcdn.net/0Rfna/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="511" data-external="1"></iframe></p> <p> <script type="text/javascript">(function(){function e(){window.addEventListener(`message`,function(e){if(e.data[`datawrapper-height`]!==void 0){var t=document.querySelectorAll(`iframe`);for(var n in e.data[`datawrapper-height`])for(var r=0,i;i=t[r];r++)if(i.contentWindow===e.source){var a=e.data[`datawrapper-height`][n]+`px`;i.style.height=a}}})}e()})();</script> </p> </section> <section class="section main-article-chapter" data-menu-title="Data ingestion and streaming"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Data ingestion and streaming</h2> <p>With traditional batch ingestion, data can be delivered too slowly for real-time analytics, agentic workflows or automated decision support, and <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-build-your-first-agentic-AI-system">schema changes can break workflows</a> without warning.</p> <p>To fill the gap, enterprises use event-driven architecture. It's a layered approach that includes several key steps.</p> <ul class="default-list"> <li><b>Change data capture.</b> CDC converts database changes into a continuous stream of events at the source.</li> <li><b>Stream processing.</b> Streaming ingests events continuously to handle stateful operations and help synchronize schema changes across systems.</li> <li><b>Validation at ingestion.</b> The final check before data reaches any users, this covers schema enforcement, nullability checks and range checks.</li> </ul> <p>There are numerous public examples of organizations that have embraced this approach.</p> <p>Netflix <a target="_blank" href="https://netflixtechblog.com/how-and-why-netflix-built-a-real-time-distributed-graph-part-1-ingesting-and-processing-data-80113e124acc" rel="noopener">built</a> a "real-time distributed graph" on an event-driven architecture when it realized batch processing could not meet the latency demands of real-time applications. Uber <a target="_blank" href="https://www.uber.com/us/en/blog/from-batch-to-streaming-accelerating-data-freshness-in-ubers-data-lake/" rel="noopener">made</a> a similar shift, replacing batch ingestion into a data lake with streaming -- a move that cut data latency from hours to minutes. Shopify <a target="_blank" href="https://shopify.engineering/2025-bfcm-live-globe" rel="noopener">runs</a> stream processing pipelines at scale to deliver real-time recommendations and buyer signals across its platform.</p> </section> <section class="section main-article-chapter" data-menu-title="Transformation and data quality"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Transformation and data quality</h2> <p>When an agent acts on the output of a poorly tested <a href="https://www.techtarget.com/searchdatamanagement/tip/The-difference-between-data-cleansing-and-data-transformation">data transformation</a>, there is no analyst to catch errors that were missed. A bad join or stale column definition can flow into model responses, retrieval results or automated decisions.</p> <p>For AI workloads, the standard is combining version-controlled transformation logic, automated quality enforcement and data contracts to reduce the risk of errors reaching AI systems. &nbsp;</p> <ul class="default-list"> <li><b>ELT vs. ETL.</b> Many modern cloud data environments favor ELT, which loads raw data first <a href="https://www.techtarget.com/data-technologies/tip/10-essential-roles-for-a-data-management-team-in-2026">and transforms it</a> for different uses in the destination platform. ELT can support flexible transformation in cloud data warehouses and lakehouses to take advantage of scalable cloud compute resources. ETL remains useful when data must be standardized, filtered or governed before it lands -- especially when bandwidth is limited, storage costs are a concern or compliance requires data masking before loading.</li> <li><b>Version-controlled transformation logic.</b> Data transformation tools, such as <a target="_blank" href="https://www.getdbt.com/blog/etl-vs-elt" rel="noopener">DBT</a>, document transformations and make them testable and auditable, with a lineage graph from raw sources to final tables. For AI workloads, this lineage should also extend to features, embeddings, model training datasets and application-facing data products.</li> <li><b>Data contracts.</b> These are <a href="https://www.techtarget.com/searchdatamanagement/opinion/Data-contracts-help-build-trustworthy-data-products-for-AI">versioned agreements between data producers and consumers</a> covering schema, data types, quality SLAs and change workflows. PayPal has open sourced its data contract <a target="_blank" href="https://github.com/paypal/data-contract-template" rel="noopener">template</a> as a reference implementation.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Orchestration and observability"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Orchestration and observability</h2> <p>Data orchestration ensures pipelines run on schedule, but a completed job does not prove that the output is correct. Data observability fills that gap, but most tooling was built for structured warehouse data and does not always <a href="https://www.techtarget.com/ai/tip/Understanding-the-limitations-and-challenges-of-RAG-systems">extend to vector stores</a> or the unstructured sources AI agents often rely on.</p> <p>The required shift is from monitoring job execution to monitoring data quality, using several components.</p> <ul class="default-list"> <li><b>Pipeline orchestration.</b> It manages scheduling, dependencies and failure recovery. Orchestration tools can handle complex pipelines that need to enforce data lineage and freshness policies.</li> <li><b>Quality-based alerting.</b> This supports monitoring of data quality issues and events. Simply documenting that a job ran is not evidence that its output is correct.</li> <li>Data observability<b>.</b> Observability platforms monitor freshness, volume, schema and distribution using anomaly detection. Leading tools have extended this to unstructured data and vector databases.</li> <li><b>Graceful degradation</b>. <a href="https://www.techtarget.com/ai/tip/Understanding-the-limitations-and-challenges-of-RAG-systems">When a pipeline stage fails</a>, serving last-known-good data and flagging partial results can help prevent incomplete or bad data from reaching downstream systems or end users. For higher-risk workflows, graceful degradation can also include quarantining suspect records, rolling back to previously validated outputs, triggering automated alerts and notifying affected users, systems or data consumers.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Access controls and data lineage"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Access controls and data lineage</h2> <p>Data lineage has long been a <a href="https://www.techtarget.com/data-technologies/tip/The-data-ownership-blind-spots-putting-organizations-at-risk">foundational aspect of data pipelines</a>, enabling users to understand where data comes from. The problem is that traditional lineage tooling often stops at the data warehouse boundary, leaving limited visibility of what data was embedded and stored in AI vector databases.</p> <p>For agentic AI data pipelines, organizations need both access control and lineage that reach the AI data retrieval layer.</p> <ul class="default-list"> <li><b>Lineage tracking.</b> Open standards let tools automatically capture and share data lineage. Column-level lineage makes it easier to see where data comes from, understand the impact of changes, and maintain audit trails.</li> <li><b>Retrieval-layer access controls.</b> Role-based and attribute-based access should be enforced at the vector store, with <a href="https://www.techtarget.com/searchsecurity/definition/personally-identifiable-information-PII">PII </a>detected and redacted during embedding.</li> <li><b>Audit logging.</b> User queries and retrieved documents must be logged to create a traceable record of what data informed each model response.</li> <li><b>Governance frameworks.</b> Frameworks such as the <a target="_blank" href="https://owasp.org/www-project-top-10-for-large-language-model-applications" rel="noopener">OWASP Top 10 for LLMs</a> and the <a target="_blank" href="https://www.nist.gov/itl/ai-risk-management-framework" rel="noopener">NIST AI Risk Management Framework</a> provide guidance on identifying risks, implementing controls and managing AI systems responsibly.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="What to do now"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What to do now</h2> <p>Batch pipelines still serve analytics workloads well, but many won't deliver the freshness, quality or traceability required for AI-driven applications.</p> <p>Use the following readiness checks to identify where a pipeline needs an upgrade:</p> <ul class="default-list"> <li>Ingestion is ready when schema changes no longer require manual intervention.</li> <li>Transformation is ready when a bad join gets caught before it reaches a model.</li> </ul> <ul class="default-list"> <li>Observability is ready when alerts fire on data quality, not just job completion.</li> <li>Governance is ready when every retrieval event has an audit trail.</li> </ul> <p><i>Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.</i></p> </section> Older pipelines may still move data, but AI workloads reveal whether they can withstand production demands, especially when agents act on data without human review. https://cdn.ttgtmedia.com/visuals/searchEnterpriseWAN/design/enterprisewan_article_002.jpg https://www.techtarget.com/data-technologies/feature/How-to-build-an-all-purpose-big-data-pipeline-architecture Tue, 18 Aug 2026 17:28:00 GMT <p>The hardest problem in enterprise AI is no longer capability. It is what happens when a proof of concept has to become a production service.</p> <p>Chief data officers have run the pilots and earned the backing. The harder task is turning a handful of working proofs into AI running across the business with a level of trust the risk committee will accept. That is where I see programs stall, and the cause is consistent. The models are ready. The data foundation beneath them is not.</p> <p>Omdia research is blunt about the gap. Only <a href="https://omdia.tech.informa.com/om138148/data-readiness-for-impactful-generative-ai">23% of organizations describe their data</a> as fully integrated for AI and analytics, and 60% say AI has made their data environment more complex. For most adopters, the technology meant to unlock enterprise data has added another layer of fragmentation.</p> <p>The pilot phase rewarded speed over structure. Every use case has its own pipeline, embeddings, copies and access exceptions. What made experimentation successful is exactly what makes production unaffordable.</p> <section class="section main-article-chapter" data-menu-title="The five pressures on AI data readiness"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The five pressures on AI data readiness</h2> <p><a href="https://www.marketingdive.com/news/how-brands-agencies-are-operationalizing-ai-as-the-tech-matures/816241/">Operationalizing AI</a> applies pressure on five fronts: speed of data readiness, cost at inference volume, controls that scale as agents begin acting on data, data trust built from lineage and freshness, and governance that is now inseparable from sovereignty and residency requirements.</p> <p>These five pressures don't fail one at a time, which is the point most evaluations miss. Programs break at the seams where two collide, where the fastest path to production violates the governance model, or where the architecture that satisfies auditors triples the inference bill. Respondents named security and compliance as the top barriers to AI agent implementation, nearly twice as often as any other factor, pinpointing those seams.</p> </section> <section class="section main-article-chapter" data-menu-title="The pattern behind successful rollouts"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The pattern behind successful rollouts</h2> <p>The market has already answered the build-versus-buy question. Ninety-one percent of respondents consider finding the <a href="https://www.techtarget.com/ai/feature/AI-agent-frameworks-A-guide-to-evaluating-agentic-platforms">right external partner</a> critical to their AI agent deployment, and 65% are partnering with AI platform providers due to implementation complexity. That is not an industry shopping for more parts. It has priced the integration tax and decided to spend its scarce talent on differentiation rather than plumbing.</p> <p>Economics closes the argument. Inference decides whether AI scales, where <a href="https://www.techtarget.com/it-strategy/feature/Tokenmaxxing-How-CIOs-can-extract-maximum-value-from-AI-tokens">cost per token</a> determines whether the tenth use case gets funded. Most <a href="https://www.datacenterknowledge.com/infrastructure/nvidia-pushes-cost-per-token-as-defining-metric-for-ai-data-centers">organizations model the cost</a> of the pilot as linear, but indexing volume and redundant reprocessing compound at a faster rate than the use case count does.</p> <p>The organizations that have crossed into production share a pattern. They <a href="https://www.techtarget.com/data-technologies/opinion/AI-doesnt-need-an-unstructured-data-dump-move-what-matters">reach data rather than move it</a>, give every data type a purpose-built home, enforce governance at the pipeline level rather than assigning it to a person, and treat the path from raw content to the production agent as a single engineered flow. The same models were available to them before. What changed was the foundation beneath them.</p> <p>Ninety-four percent of organizations say they will increase data readiness spending over the next 12 months. The intent is there. The decision about where that spending goes determines the outcomes. The enterprise AI race will not be won by the organizations with the largest models. It will be won by those who operationalize trusted data fastest.</p> <p><em>Stephen Catanzano is a senior analyst at Omdia, where he covers data management and analytics.</em></p> <p><em>Omdia is a division of&nbsp;Informa TechTarget.&nbsp;Its analysts have business relationships with technology vendors.</em></p> </section> Pilot-phase shortcuts leave enterprises with fragmented pipelines that cost too much to scale. Data foundations, rather than model capability, decide which programs reach production. https://cdn.ttgtmedia.com/rms/onlineimages/ai_g1182183209.jpg https://www.techtarget.com/data-technologies/opinion/Five-pressures-that-break-AI-programs-after-the-pilot Tue, 18 Aug 2026 10:07:00 GMT <p>Analysts and end users have sought data observability for years, but recent shifts have changed how business processes use these tools. This leaves organizations with plenty to consider when selecting a tool and deciding if commercial investment is worth it.</p> <p>One major consideration is the move from watching infrastructure to watching the data itself. Another is the arrival of GenAI and agentic workloads for consuming and transforming data products.</p> <p>Data observability tools have traditionally focused on capturing and analyzing log data to improve application performance monitoring and security. Data observability turns the focus back on the data to improve data quality, tune data infrastructure, and identify problems in data engineering pipelines and processes.</p> <p>"Data analysts and business users are the primary consumers of this data," said Steven Zhang, director of engineering at Maven Clinic. "But it's becoming increasingly common that data engineers, who produce this data alongside product engineers, are also struggling with it."</p> <p>This calls into question the trustworthiness of the data in terms of <a href="https://www.techtarget.com/searchdatamanagement/feature/Understanding-the-benefits-of-a-data-quality-strategy">accuracy, reliability and freshness</a>. This is where data observability tools come into play.</p> <p>A good data observability tool captures these problems and presents them in a clean structure. It helps consumers understand conceptually where the data went wrong and helps engineers identify the root causes.</p> <section class="section main-article-chapter" data-menu-title="Why choose a commercial tool?"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why choose a commercial tool?</h2> <p>There are many open source and commercial tools available for organizations implementing data observability workflows. Commercial tools can fast-track this process with prebuilt components for common workflows and included vendor support. They also offer increased support for enterprise use cases like <a href="https://www.techtarget.com/searchdatamanagement/news/252522412/IBM-acquires-Databand-for-data-observability-technology">data quality monitoring</a>, security and improved decision-making.</p> <p>"A modern data infrastructure is often a combination of best-in-class but disjointed set of software environments that requires to be monitored and managed in a unified manner," said Sumit Misra, vice president and business leader for data and analytics partnerships at OwlSure, an IT consultancy and services provider.</p> <p>For example, when a data job fails in one environment, another seemingly unrelated data environment must detect and react to the job's failure. Observable, responsive and self-treating <a href="https://www.techtarget.com/searchdatamanagement/feature/Best-practices-and-pitfalls-of-the-data-pipeline-process">data flows</a> are becoming essential for businesses.</p> <p>Commercial data observability tools can help organizations accelerate their time to deliver value from data quality initiatives, particularly when they are small or employ more business talent than IT talent, Misra said.</p> </section> <section class="section main-article-chapter" data-menu-title="What to look for in a data observability tool"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What to look for in a data observability tool</h2> <p>Enterprises often end up deploying more tools than they need or incorporating tools that are not specific or relevant to their business cases.</p> <p>"Investments in commercial data observability tools and initiatives need to be made from the perspective of the overall business, internal users and customers," said Alisha Mittal, a vice president in IT services at Everest Group.</p> <p>More tools do not always mean higher visibility. In fact, at times, these tools <a href="https://www.techtarget.com/data-technologies/tip/The-5-pillars-of-data-observability">increase the system's complexity</a>. Enterprises should strategically invest in observability tools by examining their current architecture, IT operations landscape and the skill development training and hiring required to handle the tools.</p> <p>Various data quality and security functions are conventionally performed by an organization's data teams. However, the value of data observability tools lies in how these activities fit into the end-to-end data operations workflow and the level of context they provide on data issues.</p> <p>Enterprises should consider how different data observability functions align with the following data quality management processes, Mittal said:</p> <ul type="disc" class="default-list"> <li>Monitoring offers a functional perspective of enterprise data systems or pipelines.</li> <li>Alerting produces alerts/notifications both for expected events and anomalies.</li> <li>Tracking provides the ability to set and track specific data-related events.</li> <li>Logging keeps a record of events in a consistent way to facilitate quicker resolution.</li> <li>Analysis involves an issue detection mechanism that provides insight on data pipelines and logs.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Agentic data observability"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Agentic data observability</h2> <p>Recent innovations in GenAI and agents are imposing new requirements on data quality and oversight. One big challenge is that they may lack the context to assess when numbers start to look absurd, requiring additional measures to assess the quality of numbers that go into subsequent processes that a human may never see directly.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> A perfectly healthy data pipeline can now produce a completely wrong business outcome. </figure> <figcaption> <strong>Yulia Plugatyreva, senior SOX and IT auditor, Chime</strong> </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>"We used to observe data. Now we need to observe decisions," said Yulia Plugatyreva, a senior SOX and IT auditor at Chime. "A perfectly healthy data pipeline can now produce a completely wrong business outcome."</p> <p>Ewelina Hayat, a consultant at technology research and advisory firm ISG, provided one example: A company might build an AI assistant over its policies and standard operating procedures, so call center employees can ask questions in natural language to deliver answers with the source and effective date attached. But when a policy changes, the document must be updated to drive updates across the entire supporting data pipeline for the new workflows, including ingestion jobs, vector embeddings enrichment, and vector database updates. But gaps in the process could mean everything appears to run correctly and the vector index job reports success.</p> <p>"But if the new document hasn't actually made it into the retrieval index, the agent can still receive yesterday's policy," Hayat said. "There is no traditional pipeline failure. Everything is green. The AI simply gives the wrong answer with confidence."</p> <p>Teams need visibility across a broader data processing estate to assess new processes that may confound agentic processes and the humans trying to make sense of their results. Vendors are beginning to ship agents as part of their platforms to diagnose and investigate problems, but caution is warranted, since automated remediation of one problem can introduce others.</p> <p>Plugatyreva's rule is to automate investigation aggressively, but to automate the action itself only when it is reversible, bounded and routine. "In fintech, 'the agent was 95% confident' is not a great explanation for why a customer-facing or financial process changed," she said.</p> </section> <section class="section main-article-chapter" data-menu-title="Top commercial data observability tools"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Top commercial data observability tools</h2> <p>Here are some of the top commercial data observability tools based on interviews with experts and users. These options are good at addressing enterprise considerations around investment, implementation and viability, Mittal said.</p> <p>They also include a well-defined, value-based approach that aligns with business goals like operational efficiency or cost savings, and have prebuilt tool stacks to help enterprises realize immediate value. These tools focus on data observability specifically and are seeing enterprise adoption.</p> <h3>Acceldata</h3> <p>Acceldata has various tools to provide data observability for cloud, hybrid, and on-premises environments. Top capabilities include data pipeline monitoring, end-to-end data reliability and quality, multilayer data observability, extensive cloud capabilities, compute and cost optimization, and rapid setup. Coverage now extends beyond database tables to unstructured and streaming data.</p> <p>The company has expanded into agentic data management. The data observability capabilities are now one module in a wider platform that also covers pipeline engineering, cataloging, governance and a runtime for AI agents. It has also added a separate line <a href="https://www.techtarget.com/data-technologies/news/366623394/Acceldata-launches-agentic-AI-powered-anomaly-detection">of AI observability capabilities</a> that trace prompts, models, retrieval steps and tool calls. Its data quality, pipeline, catalog, reconciliation and incident management functions are packaged as agents that support human in the loop workflows.</p> <p>Acceldata is good for acquiring thorough, cross-functional visibility into complicated, frequently interrelated data systems, Mittal said. This makes it the preferred observability tool in the payments and financial sector. It also excels in combining signals from many tasks and layers on a single pane of glass. This allows different teams to collaborate more efficiently.</p> <p>One caveat is these tools might not be preferable for enterprises using many different external monitoring tools, Mittal said.</p> <h3>Anomalo</h3> <p>Anomalo uses a machine learning approach to automate data observability processes to learn normal behavior and identify anomalies. This reduces the burdens on teams to write rules manually for every check. It’s aimed at organizations with large data estates and small teams that may struggle with monitoring large data pipelines in production. These capabilities extend to <a href="https://www.techtarget.com/searchdatamanagement/news/366588671/Anomalo-unveils-quality-monitoring-for-unstructured-data">monitoring unstructured data</a> with tools for evaluating document collections and scoring them for ingestion into language models for flagging problems such as missing metadata, unreadable or corrupted formats, duplication, personally identifiable information, and proprietary content. This can help build retrieval-augmented generation pipelines for internal documents.</p> <p>Recent progress has included agentic tools that automate detection, triage, and root cause analysis. This can help investigate and recommend mitigations or fixes for data engineering teams. These tools can run natively across major cloud data warehouses and lakehouses. Costs are tied to the volume of data monitored. Some FinOps discipline should be considered in scoping datasets that warrant continuous monitoring before rolling it out broadly.</p> <h3>Elementary</h3> <p>Elementary has focused on bringing data observability into data pipelines rather than adjacent to them. Its open source package can be installed into existing or new dbt projects to collect metadata, run results, and test outcomes during normal runs. These tools also support various tests for volume, freshness, distribution, and schema changes. One advantage is that these can run as native dbt tests, which simplifies workflows for versioned anomaly detection using existing <a href="https://www.techtarget.com/searchitoperations/definition/Infrastructure-as-Code-IAC">infrastructure-as-code</a> processes used by data teams.</p> <p>The open source tools can run on their own for generating reports and sending alerts via Slack and Microsoft Teams. A commercial cloud service adds column-level lineage tracking, catalog management, automated monitoring, and agent-driven anomaly investigation. It’s a good fit for teams that have standardized their data transformation pipelines on dbt.</p> <h3>IBM Data Observability by Databand</h3> <p>IBM Data Observability by Databand, which is built on technology acquired with its <a href="https://www.techtarget.com/data-technologies/news/252522412/IBM-acquires-Databand-for-data-observability-technology">purchase of Databand in 2022</a>, is a data observability platform to help teams detect and resolve data issues. One top feature is support for proactive capabilities to help detect data and resolve data incidents earlier in the development cycle. It includes tools for collecting metadata, profiling behavior, detecting and alerting on data incidents, and triaging data quality issues. It can be a good choice for companies with an extensive IBM infrastructure.</p> <p>In addition, IBM now offers data observability as a service <a href="https://www.techtarget.com/ai/podcast/IBMs-enterprise-AI-vision-for-customization-collaboration">within IBM watsonx.data integration</a>. This is part of a broader trend within the industry as data observability moves from being a standalone product to a capability built into some of the larger data engineering platforms. IBM watsonx.data integration provides a unified control plane for batch, streaming, replication and unstructured data pipelines. The data observability features monitors the health of integration flows alongside the tools that build them. AI-assisted troubleshooting capabilities help teams investigate failures and data quality problems.</p> <h3>Monte Carlo</h3> <p>Monte Carlo provides a <a href="https://www.techtarget.com/searchdatamanagement/news/252509032/Monte-Carlo-boosts-data-pipeline-observability-insights">comprehensive data observability capability</a>. It's an end-to-end platform focusing on fixing faulty data pipelines, Mittal said. It helps engineers ensure dependability and troubleshoot issues before they cause an outage. Top features include data catalogs, automated alerting and observability on several criteria. It also supports a fully automated setup.</p> <p>The company has since extended well past pipeline monitoring. It added monitoring for unstructured data such as documents, chat logs and images, without requiring users to write SQL. It has also <a href="https://www.techtarget.com/data-technologies/news/366630477/Monte-Carlos-Agent-Observability-targets-reliability-of-AI">extended data observability to AI agents</a> themselves to assess the context an agent retrieves, its behavior and tool calls, its performance and cost, and the quality of its outputs.</p> <p>In parallel with these moves, the company has begun characterizing itself as an agent trust platform, which represents an effort to move beyond the traditional data observability framing. It has also built integrations with the major cloud data platforms' agent frameworks.</p> <h3>Precisely</h3> <p>Precisely provides a <a href="https://www.techtarget.com/searchbusinessanalytics/news/252510804/Data-catalogs-fuel-increased-efficiency-speed-to-insight">data catalog</a> and data integrity suite. Its data observability capabilities help teams identify impacts from adverse data integrity issues. It focuses on ease of use, intelligent analysis that minimizes alert fatigue, and interoperability with modern tech stacks and data infrastructure. It extensively uses AI and ML capabilities to identify issues and root causes. It links data observability capabilities into a data catalog, helping teams find and integrate data into new workflows. This improves the quality of data in workflows involving enrichment with business, location or consumer data.</p> <p>Precisely has moved to package AI capabilities as assistants and agents rather than as background machine learning. The suite includes a conversational assistant. Various specialized agents <a href="https://www.techtarget.com/data-technologies/news/366634619/Precisely-intros-AI-capabilities-to-simplify-data-quality">support data quality</a>, data integration, enrichment and location intelligence workflows. These can help with configuring replication pipelines, mapping schemas and proposing quality rules. Audit trails and configuration controls support GRC requirements for regulated industries. It has also opened the suite to external AI systems, exposing its APIs through a <a href="https://www.techtarget.com/ai/tip/How-the-Model-Context-Protocol-simplifies-AI-development">hosted Model Context Protocol server</a> to allow agents and assistants built elsewhere to discover and use its data quality, enrichment and location capabilities. This combination is framed as supporting agentic-ready data.</p> <h3>Soda Data Quality</h3> <p>Soda is an AI-powered data observability platform. It also includes extensive collaboration capabilities to help data owners, data engineers and data analytics teams work through issues.</p> <p>This platform targets sophisticated data consumers. Enterprises can rapidly examine enterprise data right away, define rules to <a href="https://www.techtarget.com/searchstorage/tip/Perform-data-storage-testing-to-prevent-issues">test and validate data</a> and respond programmatically anytime a test fails.</p> <p>For instance, enterprises can immediately halt data operations and quarantine data. Checks are written in a declarative language designed for human readability that can be stored in version control with other code artifacts. The core engine is free to self-host and released under a source-available license with some restrictions compared to more permissive <a target="_blank" href="https://datakitchen.io/blog/the-2026-open-source-data-quality-and-data-observability-landscape/" rel="noopener">open source licenses</a>. The company is headquartered in Europe and offers a self-hosted runner that may appeal to European organizations concerned about data sovereignty requirements.</p> <p><i>George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.</i></p> </section> Commercial data observability tools can offer organizations pre-built components and plenty of vendor support for data use cases including monitoring, security and decision-making. https://cdn.ttgtmedia.com/rms/onlineimages/code_g1195673150.jpg https://www.techtarget.com/data-technologies/tip/7-expert-recommended-data-observability-tools Tue, 18 Aug 2026 09:38:00 GMT <p>Snowflake is doing something about the high cost of developing and managing agents and other AI applications.</p> <p>Building an agent requires workflows such as model routing, <a href="https://www.techtarget.com/data-technologies/news/366646252/Context-is-king-as-agents-evolve-but-problems-persist">context management</a>, governing tool calls, batch data processing and network orchestration that can sometimes add up to well over $100,000, according to IT consulting firm Triple Minds. As a result, it is often significantly more expensive to develop AI tools than to create the traditional data products that have historically been the primary sources enterprises used to inform decisions and manage business processes.</p> <p>With many organizations now <a target="_blank" href="https://www.gartner.com/en/newsroom/press-releases/2026-1-15-gartner-says-worldwide-ai-spending-will-total-2-point-5-trillion-dollars-in-2026" rel="noopener">spending far more</a> to build AI tools than they did to develop data products, Snowflake on Tuesday unveiled dynamic model routing in its Cortex AI Gateway control layer for AI.</p> <p>Dynamic model routing aims to reduce wasted spending on using bigger, more expensive models than are needed to inform many agents and instead automate model selection to strike the optimal balance between the quality of the model and <a href="https://www.techtarget.com/enterprise-software/feature/AI-feature-spend-is-the-new-software-cost-control-problem">the cost of the project</a>.</p> <p>"These capabilities are highly significant because they attack the operational overhead and suboptimal selection of models," William McKnight, president of McKnight Consulting, told TechTarget. "By automatically matching task complexity to the optimal model, dynamic routing eliminates the need for developers to rebuild their application infrastructure every time a new model is released."</p> <p>Beyond dynamic model routing, Snowflake revealed that it plans to add models from <a href="https://www.techtarget.com/whatis/feature/DeepSeek-explained-Everything-you-need-to-know">DeepSeek</a> and Z.ai to <a href="https://www.techtarget.com/data-technologies/news/366625218/Snowflake-continues-to-add-AI-boost-Cortex-capabilities">its Cortex AI development environment</a>. The vendor already enables access to models from Anthropic, Google, Mistral, OpenAI and SpaceX. More model choices better enable users to strike the optimal balance between cost and performance.</p> <p>Dynamic model routing and access to Z.ai's GLM-5.3 are not yet in private preview, while access to DeepSeek-V4-Flash is in private preview.</p> <section class="section main-article-chapter" data-menu-title="Customers want cost control"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Customers want cost control</h2> <p>AWS in December made cost control the focal point of <a href="https://www.techtarget.com/data-technologies/news/366635663/Latest-AWS-data-management-features-target-cost-control">its data management initiatives</a> given that AI development demands massive data workloads. Databricks and Microsoft also offer capabilities aimed at helping customers control their spending on AI.</p> <p>Now, Snowflake is similarly addressing the cost of AI development and deployment.</p> <p>As more enterprises move AI pilots into production, they're paying more attention to spending than they did when they were just trying to prove that something worked, according to Sanjeev Mohan, founder and principal analyst of analyst firm SanjMo. That is the reason data and AI providers are beginning to prioritize cost control.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Companies run AI across many apps and agents, and the default habit of sending every request to the most powerful frontier model turns out to be enormously wasteful, since most tasks don't need that horsepower. </figure> <figcaption> <strong>Sanjeev Mohan</strong>Founder and principal analyst, SanjMo </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>"Companies run AI across many apps and agents, and the default habit of sending every request to the most powerful frontier model turns out to be enormously wasteful, since most tasks don't need that horsepower," he told TechTarget.</p> <p>For example, Canva was <a target="_blank" href="https://finance.yahoo.com/technology/ai/articles/canva-rare-startup-grew-fast-201427317.html" rel="noopener">forced to cut its growth forecast</a> from 30% to 20% because it relied too heavily on frontier models that proved more costly than expected, Mohan continued.</p> <p>"The analyst read is that AI inference breaks the near-zero marginal cost that made software so profitable. … Managing spend across models is now worth billions," he said.</p> <p>McKnight likewise suggested that as enterprises move more AI pilots into production, cost is becoming <a href="https://www.techtarget.com/it-strategy/feature/What-Big-Techs-AI-spending-means-for-your-IT-budget">a greater concern</a>, and vendors must respond to keep their customers.</p> <p>"Enterprises are under pressure to transition speculative AI pilots into ROI-generating production and contain runaway cloud compute costs and the extreme engineering overhead of multi-vendor stacks," he said.&nbsp;"To prevent customers from abandoning these initiatives, database and orchestration vendors like Snowflake are prioritizing predictable pricing models that make AI development financially sustainable."</p> <p>Customers, in fact, provided Snowflake with the impetus to develop dynamic model routing and expand the selection of models its users can choose from when building AI tools, according to Pavan Pothukuchi, the vendor's director of product for AI and machine learning.</p> <p>He noted that with customers "becoming more rigorous about the economics" as AI <a target="_blank" href="https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html?id=us:2ps:3gl:aisgm26:awa:CONS:em:K0218784:012626:kwd-430833501819:195648817121:794247818306::&amp;gclsrc=aw.ds&amp;gad_source=1&amp;gad_campaignid=23269751971&amp;gclid=CjwKCAjwhZDUBhBGEiwAbi5bjpnKWPhwQ_x2iRpG4nnYjNKAl3JOosCPrbhIeSMsOTahBvicfzbySRoCS8MQAvD_BwE" rel="noopener">moves from experimentation to production</a>, Snowflake saw an opportunity to help them reduce spending by optimizing model selection.</p> <p>"Customers weren't necessarily asking us specifically for dynamic model routing, but they were asking us to help improve the economics and performance of their AI deployments," Pothukuchi said. "Dynamic model routing is a natural response to that … so customers can focus on the outcomes they want from AI."</p> </section> <section class="section main-article-chapter" data-menu-title="Addressing AI spending"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Addressing AI spending</h2> <p>Snowflake introduced Cortex AI Gateway in July 2025 to provide administrators with a single location to govern connections and interactions between agents, intelligently route workloads, and optimize AI use. Within Cortex AI Gateway, users can view <a href="https://www.techtarget.com/whatis/definition/token">token</a> usage and costs, and establish spending limits for agents and other AI tools so they don't unexpectedly drive up costs.</p> <p>The addition of dynamic model routing helps customers control their AI spending by automating the selection of the models that agents are built upon.</p> <p>Snowflake noted that using the same model for every task can drive up costs since certain models are better suited for an agent's specific task than others. For example, one model may be better suited for informing <a href="https://www.techtarget.com/enterprise-software/feature/Early-adopters-of-agentic-AI-for-customer-service-offer-advice">customer service agents,</a> while another may be better suited for <a href="https://www.computerweekly.com/feature/Forrester-Managing-supply-chain-volatility-with-agentic-AI">supply chain optimization agents</a>.</p> <p>Dynamic model routing automatically funnels work that necessitates deep reasoning to frontier models while directing lower complexity tasks to less expensive models. In addition, routing decisions update as models and their pricing change, so users don't have to rebuild agents and other AI applications to lower spending.</p> <p>"Dynamic model routing directly addresses the primary bottleneck of enterprise AI, which is the operational burden of manual model selection at scale and automatically matching each workload to the optimal AI model based on quality, speed, and cost," McKnight said.</p> <p>Mohan likewise noted that the new feature directly addresses a substantial problem many enterprises face.</p> <p>"The routing … removes a real operational chore, and does it automatically inside the platform where the data already lives," he said.</p> <p>While dynamic model routing addresses customers that want to control costs, it does not distinguish Snowflake from its main competitors, according to Mohan. He noted that Microsoft offers Model Router in <a href="https://www.techtarget.com/ai/news/366616024/Microsoft-intros-Azure-AI-Foundry-for-building-AI-apps">AI Foundry</a>, Databricks provides routing through Unity AI Gateway, and vendors including OpenRouter and Vercel automate model routing.</p> <p>However, its deep integration with Snowflake's <a href="https://www.techtarget.com/data-technologies/news/366643795/Snowflake-barrage-adds-more-AI-development-analysis-tools">broader data and AI platform</a>, including tools such as CoCo and CoWork, along with governance capabilities such as role-based access, does provide differentiation.</p> <p>"What's different about Snowflake's version is where it sits, routing inside Snowflake's governed boundary," Mohan said. "For a Snowflake-centric enterprise, routing that never moves data outside governance and attributes spending to the right team is the distinction, not the routing algorithm itself."</p> <p>Meanwhile, Snowflake's overall cost of building and managing AI workloads is competitive, according to McKnight. Capabilities such as real-time concurrent LLM serving with a 0% error rate, optimized compute sizing, and Snowflake <a href="https://www.techtarget.com/it-infrastructure/definition/What-is-container-management-and-why-is-it-important">Container Services</a> aid its price-performance ratio.</p> <p>"Against the native services of major cloud providers, Snowflake is a highly competitive, well-balanced option for search," McKnight said.</p> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineimages/gen_ai_costs_beyond_development-h.png"> <img data-src="https://www.techtarget.com/rms/onlineimages/gen_ai_costs_beyond_development-h_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineimages/gen_ai_costs_beyond_development-h_mobile.png 960w,https://www.techtarget.com/rms/onlineimages/gen_ai_costs_beyond_development-h.png 1280w" alt="AI costs extend beyond the initial feature price. Change management, infrastructure, model run costs, governance and ongoing optimization all affect the real cost of enterprise AI. " data-credit="TechTarget"> <figcaption> <i class="icon pictures" data-icon="z"></i>With model routing among the reasons AI development and deployment costs are spiraling for some organizations, Snowflake introduced a feature that automates model selection to optimally balance cost and performance. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="Looking ahead"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Looking ahead</h2> <p>Over the final months of 2026, Snowflake's plans include improving how the pieces that comprise <a href="https://www.techtarget.com/ai/tip/Agentic-AI-workflows-Trends-examples-and-best-practices">the AI stack</a> work together, according to Pothukuchi. Regarding dynamic model routing, he added that making the feature more adaptive will be a priority.</p> <p>"We want to get increasingly sophisticated at understanding what a particular customer is trying to accomplish and selecting the intelligence best suited to that task, while continuously optimizing for quality and efficiency," Pothukuchi said.</p> <p>Mohan suggested that Snowflake could build on dynamic model routing by adding cost control tools that go beyond routing to include generating and acting on their own recommendations.</p> <p>"The natural next step is … moving from control to active guidance, which is exactly the muscle companies like Canva had to build by hand," he said.</p> <p>McKnight, meanwhile, advised Snowflake to take steps to address the complexity of <a href="https://www.techtarget.com/data-technologies/news/366641473/Snowflake-broadens-open-source-embrace-ups-Iceberg-support">its platform</a> and opaque billing that can catch customers off guard.</p> <p>Specifically, it could turn its open-source pg_lake extension into a fully managed rival of <a href="https://www.techtarget.com/data-technologies/news/366638723/Databricks-launches-PostgreSQL-Lakebase-to-aid-AI-developers">Databricks' Lakebase,</a> natively integrate an open semantic layer with Cortex AI and its Horizon Catalog to securely govern data mesh architectures, and add AI-driven <a href="https://www.techtarget.com/ai/tip/How-to-apply-FinOps-to-optimize-agentic-AI-costs">FinOps controls</a> along with micro-concurrency scaling to make spending more predictable.</p> <p>"To continue holding its prime position in the market, Snowflake must focus on eliminating architectural complexity and billing anxiety," McKnight said.</p> <p><i>Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.</i></p> </section> With spending on AI spiraling out of control for many organizations, the vendor's latest new feature automates model selection to balance cost and performance. https://cdn.ttgtmedia.com/rms/onlineimages/money_g1050046190.jpg https://www.techtarget.com/data-technologies/news/366649521/Snowflake-targets-cost-of-AI-with-dynamic-model-routing Tue, 18 Aug 2026 09:07:00 GMT <p>As of Aug. 2, the EU AI Act assesses penalties for the first time against AI oversights stemming from poor data management. The Act places the onus of compliance squarely on data teams and applies to any organization that interacts with EU residents.</p> <p>A clear problem is that most organizations didn't institute their data infrastructure with such documentation in mind because it wasn't previously required. Now that the Act's obligations for transparency, general purpose AI (GPAI) models and AI literacy are in force, there's a divide between what it penalizes and where most data governance programs actually stand.</p> <p>By examining which specific data management mechanisms are required and where most data governance programs fall short, data teams can identify concrete actions to rectify these shortfalls.</p> <section class="section main-article-chapter" data-menu-title="Data management requisites"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Data management requisites</h2> <p>Originally, the AI Act mandated high-risk Annex III systems to be fully compliant by August 2, 2026. The <a href="https://www.techtarget.com/searchenterpriseai/news/366646620/EU-AI-Act-compliance-deadline-is-here-What-to-watch">May 2026 Digital Omnibus</a> agreement tabled many of those requirements until December 2027. The components that took effect Aug. 2 still demand rigorous data management.</p> <p>Under Article 50's transparency requirements, data teams must mark AI-manipulated depictions of real people and deepfakes as AI-generated. Chatbots and similar conversational applications must inform people they're interacting with AI, unless it's obvious, and users must be told when biometric categorization or emotion recognition is in use.</p> <p>GPAI model providers, including teams who substantively tailor foundation models or build applications heavily reliant on them, must keep technical documentation under Annex XI, <a href="https://www.techtarget.com/ai/tip/Explore-the-role-of-training-data-in-AI-and-machine-learning">furnish training data summaries</a>, and confirm the data adheres to EU copyright law under a documented policy. Data literacy is also required of all users and operators of AI systems.</p> </section> <section class="section main-article-chapter" data-menu-title="Governance shortcomings"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Governance shortcomings</h2> <p>For data teams, the biggest obstacle to EU AI Act compliance is AI governance itself. <a href="https://www.gartner.com/en/documents/7854881">A 2025 Gartner survey</a> on how organizations handle cybersecurity risk found that nearly 90% lacked AI governance programs. Although that figure represents only one slice of the enterprise, it still indicates how immature AI governance is today. Many organizations run AI governance tied to specific employees or projects. Meeting the Act's transparency and GPAI mandates becomes far easier when the roles, rules and responsibilities for governing AI systems are <a href="https://www.techtarget.com/data-technologies/opinion/Data-governance-fails-without-CFO-ownership">formalized, documented and spread</a> throughout the enterprise.</p> <p>Many users and operators of AI systems know their own tooling but <a href="https://www.techtarget.com/ai/tip/How-to-keep-data-silos-from-damaging-your-AI-projects">not what runs in other departments</a> or serves other customers. Compliance with the Act's GPAI and transparency requirements all but requires data teams to maintain a comprehensive inventory of which AI systems the organization operates, along with their data dependencies and formal documentation of both -- a demand that falls hardest on GPAI model providers.</p> <p>Noncompliance penalties for GPAI models are €15M or 3% of global turnover, whichever is higher. Systemic risk GPAI models are trained with 10²⁵ FLOPs of compute and require adversarial testing such as red teaming, cybersecurity protections and energy consumption data disclosures. Data teams without complete inventories of AI assets can't meet this requirement.&nbsp;&nbsp;&nbsp;&nbsp;</p> <p>Siloed, employee-tied governance undermines transparency even more directly when it comes to labeling model outputs as AI-generated. When only individual employees hold that knowledge, it rarely reaches end users, even if they pass it to a manager or colleague. The Act requires that knowledge be readily available to the general public when people interact with these systems, as well as to internal users. Noncompliance penalties are €7.5M or 1.5% of global turnover, whichever is higher.</p> </section> <section class="section main-article-chapter" data-menu-title="Remediation"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Remediation</h2> <p>Applying the fundamentals of data governance to their AI instances lets data teams close these gaps. The first step is to classify AI assets fully: the models themselves, their use and users, training data, data models, taxonomies, and data sources. That inventory is what the Act's GPAI documentation and transparency obligations assume already exists. Without it, a data team cannot show which systems are in scope, let alone document them.</p> <p>Typically, only a few key employees hold this institutional knowledge. However, data teams can formalize it through enterprise architecture to map these systems, and through regex, machine learning, and other statistical methods to discover, classify and tag them.&nbsp;</p> <p>Next, data teams should categorize the AI assets by <a href="https://axis-intelligence.com/eu-ai-act-enforcement-guide/">the EU AI Act risk tier</a>: minimal risk, limited risk with transparency requirements, high risk, and prohibited. The tier a system falls into determines which obligations attach to it. A limited-risk chatbot triggers Article 50's disclosure rules, while a systemic GPAI model pulls in adversarial testing and energy reporting. These tiers inform next steps, particularly the immediate documentation of GPAI models and public disclosures when people are interacting with AI systems.</p> <p>The tiers require data teams to update existing governance policies or write new ones. Splitting those policies into two distinct sets helps: one set for the public's interactions with AI systems, another for internal work, such as <a href="https://www.techtarget.com/whatis/definition/red-teaming">red teaming</a> or cybersecurity for systemic GPAI models. Legal counsel can tell an organization whether their use of GPAI models classifies it as a model provider, even if it didn't build the model.</p> <p>In many cases, deployments of dynamic agents can automate parts of the transparency requirements, including <a href="https://www.techtarget.com/searchenterpriseai/feature/How-watermarking-AI-content-benefits-businesses">watermarking content as AI-generated</a> or triggering workflows to inform end users they're interacting with AI.</p> <p><i>Jelani Harper is a data industry analyst and journalist covering data management,&nbsp;AI&nbsp;and enterprise IT for more than a decade. He is a research&nbsp;lead&nbsp;at Blue Badge Insights, writing analyst reports for&nbsp;GigaOm&nbsp;and articles for VentureBeat and The New Stack.&nbsp;&nbsp;</i>&nbsp;</p> </section> Most enterprises govern AI in siloed, employee-tied pockets, but the EU AI Act assumes formalized enterprise-wide governance. That gap drives the risk data teams now face. https://cdn.ttgtmedia.com/rms/onlineimages/legal_g1065824400.jpg https://www.techtarget.com/data-technologies/feature/Where-data-teams-fall-short-on-governance-for-the-EU-AI-Act Mon, 17 Aug 2026 09:30:00 GMT <p>AI makes it look effortless to spin up BI application prototypes and reports, which gives some enterprise leaders the false impression that platform engineering is just as easy.</p> <p>To illustrate the first point, take a common scenario: a data analyst needs a customer churn breakdown by region for a meeting in two hours. Instead of filing a ticket with the BI team or building the report herself in the company's self-service BI tool, she <a href="https://www.techtarget.com/data-technologies/post/Why-the-rush-to-replace-dashboards-with-AI-is-a-mistake">asks a coding agent to build it</a>. It comes back wired to the data warehouse, filtered and ready before the meeting starts. That's a point solution: a report built to answer one question -- and AI just made that process dramatically faster than either path the analyst previously had.</p> <p>Over the past 20 years, self-service BI solved a real problem. It lets people who aren't software engineers query the warehouse and build their own reports without waiting on a BI team. But it works inside the tool's boundaries: whatever charts, filters and layouts the BI platform allows. AI removes that boundary. The visuals, interactions and underlying logic are now buildable from scratch, styled and structured however someone wants, no longer constrained by what a BI tool's feature set supports.</p> <p>That matters because no BI tool ever fully satisfies its users. There's always a report that needs a workaround or a data visualization the platform wasn't designed for. I've seen this complaint in every organization I've been part of, in one form or another.</p> <p>But there's a downside to freeing users from the old constraints. Self-service BI already <a href="https://www.techtarget.com/data-technologies/feature/How-dashboard-sprawl-challenges-upend-enterprise-analytics">led to a proliferation of user-built reports</a>. AI adds another layer on top: allowing more report builders to create their own answers, with different logic and definitions behind visuals that look like they're answering the same question but aren't.</p> <p>This unchecked expansion is part of the problem, too. It creates exactly the kind of mess a single, standardized system is supposed to prevent: conflicting answers on what looks like the same question. What's often missing is a <a href="https://www.techtarget.com/data-technologies/feature/Why-enterprise-AI-depends-on-the-semantic-layer">shared semantic layer</a> that defines what the numbers mean, plus the governance to make existing reports discoverable and retire the ones nobody trusts anymore.</p> <p>That gap makes the next question feel reasonable: if a coding agent can build the report this fast, why not have it build a full platform, at a fraction of what it costs to buy commercial tools like Tableau or Looker? Companies are already taking this kind of approach in categories with a much bigger lift than BI. Salesforce's price increases have pushed some organizations toward building their own CRM logic. If that's happening with CRM, a BI platform can look like an easier version of the same move.</p> <p>But that initial estimate only counts the build. It doesn't count what comes after -- and the CRM comparison doesn't hold up as well when you do. A homegrown CRM carries real overhead, too, but it can make sense when the licensing fees for commercial software are expensive enough and heavy customization would be required for your business processes anyway.</p> <p>A BI platform doesn't have that same business case. It isn't tied to any unique business process, unlike a CRM that might encode a specific sales team's approval chains or territory rules. <a href="https://www.techtarget.com/data-technologies/feature/9-examples-of-business-intelligence-use-cases-for-companies">Querying and visualizing data</a> is largely the same at every company, meaning that building and maintaining a custom BI platform is rarely a differentiator.</p> <p>Set the resources issue aside. A harder problem remains, no matter who or what builds the BI platform. Creating a single report is genuinely simple now; a coding agent did exactly that in two hours. Productizing a platform that an entire organization can rely on, across current and future use cases, is entirely different work, and AI only makes the first task faster.</p> <p>Open source BI set out to fix what people didn't like about commercial platforms, promising something simpler, cheaper and more customizable. Superset, built at Airbnb and still maintained as an open source project, later became the foundation for Preset's hosted offering. Metabase took a different route, leaning into usability with a no-code query builder that non-technical people could actually use.</p> <p>Both addressed real problems: licensing cost, vendor lock-in and customization limits. But neither removed the tradeoff entirely. Superset still requires teams to own hosting and maintenance if they run it themselves. And neither tool matched the breadth of dashboard types, integrations and enterprise access control that the commercial BI platforms had built over the years.</p> <p>Query performance has to survive an audit months after launch, not just hold up on day one. Someone needs to catch failures before users report them, and there needs to be a roadmap for what the platform will look like a year out, with a feedback loop to drive improvements. That's what productizing requires, and none of it gets easier because a prototype comes together fast.</p> <p>The upfront cost comparison part of the pitch usually compares a licensing fee against the first working version: the prototype a coding agent produces in an afternoon. That comparison leaves out the more expensive part: turning a prototype into <a href="https://www.techtarget.com/searchenterpriseai/post/How-CIOs-should-architect-trust-in-AI-not-just-govern-it">something a wide user base can trust</a>, with proper access control, high availability and an active maintenance plan. That productization phase typically requires far more capital and labor than it took to get the first working version. This gap is exactly where "build our own platform" math quietly stops counting.</p> <p>None of that makes build the right call. What's changed is that AI makes initial development look so easy that companies stop checking whether buying was actually the problem. Was it too expensive, too complex or insufficient for the need, even after factoring in what the tool could be extended or configured to do?</p> <p>A BI and analytics leader can run the decision through three questions:</p> <ul class="default-list"> <li><b>Core vs. commodity.</b> Is this tied to something core to how the business operates, or is it infrastructure that no company gets paid to run?</li> <li><b>Total cost of ownership.</b> Does the cost model include what it takes to run this as a product long after it ships: security, reliability, monitoring, a lifecycle plan and a feedback loop to make improvements?</li> <li><b>Organizational capability</b>. Is that ongoing work getting treated as the different discipline it actually is, rather than as more coding? That takes its own skills and its own organizational muscle, whether it sits with an internal team or an outside partner.</li> </ul> <p>Point solutions are still fine to build. A single report, a specific app, a one-off analysis with a known shelf life: build it fast and retire it when the question's answered. But the platform is a different decision, and it deserves scrutiny that has nothing to do with how fast the prototype came together.</p> <p><i>Sireesha Pulipati is a data and AI engineering leader specializing in scalable data platforms, real-time data pipelines and production-grade AI systems. She is a staff data engineer at Shopify and previously worked at Google, where she led analytics and business intelligence across search and Google Cloud. The views expressed by Pulipati are her own and do not reflect those of her company.</i></p> AI makes generating BI reports simple, which makes platform-building look easier than it is. BI leaders should weigh the long-term commitment before deciding to build. https://cdn.ttgtmedia.com/rms/onlineimages/code_g1127196618.jpg https://www.techtarget.com/data-technologies/opinion/Why-AI-for-BI-works-for-reports-but-not-platform-building Fri, 14 Aug 2026 15:08:00 GMT <p>Databricks keeps raising funding.</p> <p>After revealing in July that it was in the process of securing new financing, Databricks on Thursday closed on $5 billion in venture capital funding at a rate that values the company at $190 billion and brings its total financing over $32 billion.</p> <p>But after raising $5 billion or more in four separate funding rounds -- <a href="https://www.techtarget.com/data-technologies/news/366617434/Databricks-record-10-billion-funding-round-to-fuel-growth">including $10 billion in December 2024</a> -- the question isn't what the data management and AI vendor should do with the money or whether it can raise more. Instead, it is what Databricks should do next.</p> <p>While many data management and analytics vendors are struggling to attract investors, there seems to be an endless supply of private capital willing to invest in Databricks. Beyond rounds exceeding $5 billion, the vendor raised $2 billion in debt financing in February and added $1 billion or more three additional times.</p> <p>However, an eventual initial public stock offering would be more advantageous for Databricks than continuing to raise private funding, according to Michael Ni, an analyst at Constellation Research.</p> <p>"Staying private has allowed Databricks flexibility to ignore public scrutiny on its profitability and predictability, but a successful public IPO could provide a growth flywheel," he told TechTarget.</p> <p>Specifically, if the public markets respond positively to Databricks' <a href="https://www.techtarget.com/data-technologies/news/366644813/Databricks-intros-new-Genie-data-management-tools-to-aid-AI">continued additions</a> of &nbsp;data and AI stack components, a healthy stock price would give the vendor greater flexibility to expand than raising private funding, he continued. In addition, going public -- and the need to respond to public-market sentiments -- would make it easier for the CIOs of potential customers to make Databricks part of their AI and <a href="https://www.techtarget.com/data-technologies/definition/What-is-data-architecture-A-data-management-blueprint">data architectures</a>.</p> <p>"The value in an IPO would go beyond the cash raised, enabling Databricks to move from a successful private platform to a consolidator and long-term standard for the enterprise AI stack," Ni said.</p> <p>However, when an IPO happens depends, to some degree, on external factors such as the <a target="_blank" href="https://www.ey.com/en_us/insights/ipo/ipo-market-trends" rel="noopener">market conditions for major IPOs</a>. Also in question is how significantly an IPO would benefit Databricks when the company is seemingly able to raise funding whenever it desires to fuel technological innovation, acquisitions and other initiatives.</p> <section class="section main-article-chapter" data-menu-title="Databricks is different"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Databricks is different</h2> <p>Databricks is far from the only data management vendor that has expanded over the past few years to provide a data foundation for AI development.</p> <p>Rival <a href="https://www.techtarget.com/data-technologies/news/366643795/Snowflake-barrage-adds-more-AI-development-analysis-tools">Snowflake</a> and hyperscale cloud providers <a href="https://www.techtarget.com/data-technologies/news/366644853/AWS-latest-to-introduce-context-layer-for-agentic-AI">AWS</a>, <a href="https://www.techtarget.com/data-technologies/news/366641929/Google-unveils-data-cloud-purpose-built-for-agentic-AI">Google Cloud</a> and <a href="https://www.techtarget.com/data-technologies/news/366643955/Microsoft-boosts-Fabric-to-make-it-a-foundation-for-AI">Microsoft</a> have all similarly added capabilities that enable customers to connect AI tools with proprietary data. In addition, database vendors such as MongoDB and Couchbase, niche specialists including Alation and Informatica, and analytics providers such as Qlik and Tableau have all added tools to enable AI development <a href="https://www.techtarget.com/data-technologies/news/366646252/Context-is-king-as-agents-evolve-but-problems-persist">informed by relevant, high-quality data</a>.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> The value in an IPO would go beyond the cash raised, enabling Databricks to move from a successful private platform to a consolidator and long-term standard for the enterprise AI stack. </figure> <figcaption> <strong>Michael Ni</strong>Analyst, Constellation Research </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>Databricks' valuation of $190 billion, however, dwarfs Snowflake's market capitalization of $114 billion, despite Snowflake's stock price more than doubling since April. And while Databricks continues to raise billions of dollars with each funding round, vendors such as Confluent, Domo, Dremio, DBT Labs and Fivetran have either sought buyers or merged with peers with funding difficult for data and analytics vendors to raise since <a href="https://www.techtarget.com/data-technologies/news/252520740/Tech-stock-sell-off-signals-tough-times-for-data-vendors">tech stocks plummeted in 2022</a>.</p> <p>Venture capitalists simply view Databricks differently than they view most other data and AI providers, according to Sanjeev Mohan, founder and principal of analyst firm SanjMo.</p> <p>"Investors see Databricks becoming the single consolidated platform for a wide span of data and AI use cases," he told TechTarget. "Databricks is different in that it actually keeps absorbing adjacent categories onto one architecture rather than bolting on a feature and calling it a platform."</p> <p>In addition, venture capitalists have faith in Databricks CEO Ali Ghodsi, Mohan continued.</p> <p>"Investors find their CEO to be a force of nature who has tremendous ambition and grit to build out the platform," he said.</p> <p>One of those investors is Benjamin Black, <!--StartFragment--><span data-olk-copy-source="MessageBody">co-founder and CIO of Powerlaw Capitals Group, which holds exposure to Databricks through Powerlaw Corp. </span></p> <p>Numerous factors have led investors to flock to Databricks while only minimally funding some of its smaller competitors, according to Black. In particular, he noted that its execution at scale -- surpassing $7 billion in annual recurring revenue and year-over-year revenue growth of approximately 80% -- are significant factors in the venture capital community's <a href="https://www.techtarget.com/data-technologies/news/366617350/Record-funding-round-reflects-Databricks-differentiation">continued interest in Databricks</a>.</p> <p>"Databricks is popular because it has demonstrated unusually strong execution at enormous scale just as enterprises are trying to turn AI from a promising experiment into a core business capability," Black said.</p> <p>Strong growth at scale and technology execution -- a platform designed to enable users to successfully develop AI tools -- makes Databricks not merely a company investors want as part of their portfolio, but one they want to get in on early, he continued.</p> <p>"It has the revenue, growth, and institutional maturity to list when it chooses," Black said. "That combination is precisely why investors want exposure now rather than waiting for a ticker symbol.</p> <p>Beyond how investors view Databricks, which is as much a bet on growth potential and financial performance than a recognition of technological prowess, Databricks has distinguished itself from a technological perspective as well, according to David Menninger, an analyst at ISG Software Research.</p> <p>Significant features Databricks has developed over the past few years include Unity Catalog for governing data and AI, Mosaic AI for developing generative and agentic AI, <a href="https://www.techtarget.com/data-technologies/news/366638723/Databricks-launches-PostgreSQL-Lakebase-to-aid-AI-developers">Lakebase</a> to provide a PostgreSQL database foundation for AI initiatives, <a href="https://www.techtarget.com/data-technologies/news/366625695/Latest-Databricks-tools-use-AI-to-simplify-AI-development">Agent Bricks</a> to automate aspects of AI development, and its Genie natural language interface.</p> <p>"Databricks has executed well," Menninger said. "Investors are primarily interested in high-growth companies. Databricks has been growing rapidly for many years."</p> <p>ISG groups Databricks with 12 other data and AI platform providers, he continued. Peers include fellow data platform providers Cloudera, Snowflake and Teradata, <a href="https://www.techtarget.com/it-infrastructure/definition/What-is-hyperscale-cloud-Computing-and-data-center-uses-explained">hyperscale cloud</a> vendors AWS, Google Cloud and Microsoft, and other broad-based technology providers such as IBM, Oracle and SAP.</p> <p>"We rate Databricks as an overall leader among this group, which means we place them in the top three," Menninger said.</p> </section> <section class="section main-article-chapter" data-menu-title="Private vs. public"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Private vs. public</h2> <p>In theory, Databricks could stay private for the foreseeable future.</p> <p>"Databricks has had virtually no financial reason to face the public markets because the private markets are treating them so favorably," William McKnight, president of McKnight Consulting, told TechTarget.</p> <p>Similarly, Mohan noted that private investors have been so willing to keep Databricks flush with cash that the vendor has the luxury of waiting until the ideal time to <a href="https://www.techtarget.com/data-technologies/news/366636532/Databricks-adds-4B-funding-round-IPO-could-be-next">explore an IPO</a>.</p> <p>"The private market has given Databricks everything an IPO would, without the constraints," he told TechTarget.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> Private secondary sales only stretch so far. A public market lets employees and early backers actually cash out, unlike private market. That is good for employee retention and morale. </figure> <figcaption> <strong>Sanjeev Mohan</strong>Founder and principal analyst, SanjMo </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>Eventually, however, the benefits of going public outweigh those of remaining private, Mohan continued.</p> <p>For example, each private funding round dilutes the value that existing investors hold in Databricks. In addition, public companies have the transparency and credibility that can attract large enterprise and government customers.</p> <p>"Private secondary sales only stretch so far," Mohan said. "A public market lets employees and early backers actually cash out, unlike the private market. That is good for employee retention and morale."</p> <p>McKnight likewise noted that the <a href="https://www.investopedia.com/ask/answers/021015/what-difference-between-ipo-and-private-placement.asp">benefits of going public</a> ultimately outweigh those of continuing to raise private capital. Specifically, he noted that an IPO would enable Databricks to have access to liquid, market-validated funds for future acquisitions, stock to attract and retain elite talent, access to cheap debt and secondary capital markets, and SEC-audited transparency.</p> <p>"Even with unlimited private capital, going public provides advantages that private funding cannot replicate," McKnight told TechTarget.</p> <p>That said, raising more private funding remains a possibility for Databricks.</p> <p>Black noted that a company such as Databricks can theoretically remain private indefinitely, as long as it can continue to raise capital, its employees and early investors can get liquidity without the company going public, and it can demonstrate its financial health without public filings.</p> <p>"Private capital has gotten deep enough, with the largest funds able to write enormous checks, that a company can keep raising privately for a long time without ever needing the public markets," Black said.</p> <p>In fact, it's possible that Databricks and other companies,such as <a href="https://www.tradingview.com/symbols/NASDAQ-STRIPE/">Stripe,</a> that are able to raise massive amounts of funding, could create a new class of "forever private" companies, he continued.</p> <p>"Databricks and Stripe get all primary and secondary liquidity they need in the private markets," he said, noting that they can innovate without having to meet the expectations of public markets and each has already made the type of acquisitions that public stock is often used to fund. "If the private markets stay so accommodating for the best mega-privates, why go public at all?"</p> <p>Menninger likewise noted that Databricks -- in theory -- could keep raising funding and remain private. He pointed out that IPOs typically fund expansion, and Databricks has been able to expand by <a href="https://www.techtarget.com/data-technologies/news/366623864/Databricks-adds-Postgres-database-with-1B-Neon-acquisition">making numerous acquisitions</a> and investing in product development.</p> <p>"If you have access to the capital you need, it’s much better not to have to face the quarterly scrutiny of the public markets. It can really distract from the day-to-day operations of the business," Menninger said.</p> <p>However, the greater likelihood is that Databricks will eventually go public, he continued, noting that benefits such as the public disclosure of financials attracting new customers, equity given to employees having more value when a company is public, and more easily funded acquisitions could outweigh the benefits of remaining private.</p> <p>"Investors need to be able liquidate their investments at some point," Menninger said. "As long as there are other investors willing to step in, they can potentially go on like this forever, but that's highly unlikely. Even if they reach the point where they don't need additional capital, the existing investors will want to be able to close out their positions."</p> </section> <section class="section main-article-chapter" data-menu-title="If not now, when?"> <h2 class="section-title"><i class="icon" data-icon="1"></i>If not now, when?</h2> <p>Databricks will eventually go public, according to Ghodsi.</p> <p>"We will go public, I promise," he said during an interview with CNBC on Thursday following the closing of Databricks' $5 billion funding round.</p> <p>But the timing for an IPO has to be right, Ghodsi emphasized.</p> <div class="imagecaption alignLeft"> <img src="https://cdn.ttgtmedia.com/rms/onlineimages/ghodsi_ali.jpg" alt="Databricks CEO Ali Ghodsi">Ali Ghodsi </div> <p>The stock market, though there have been significant IPOs in 2026, has been volatile, <a target="_blank" href="https://www.google.com/search?q=dow+jones&amp;oq=dow+jones&amp;gs_lcrp=EgRlZGdlKgYIABBFGDkyBggAEEUYOTIICAEQ6QcY_FXSAQgxODE2ajBqNKgCAbACAQ&amp;sourceid=chrome&amp;ie=UTF-8" rel="noopener">trending generally up</a> but lurching more in response to economic indicators, major companies' earnings filings and other reports.</p> <p>"The markets have been really wobbly," Ghodsi told CNBC. "For three months, everything is down and it's going to be terrible. And then the other day, it's amazing. There was one great [announcement] by one software company and the whole market is up 10%, and then the next day everything is down 10%. It's just very turbulent times."</p> <p>Beyond market unpredictability, the final months of 2026 may not be a good time for Databricks to go public because other major tech companies are also planning to go public and a Databricks IPO could get overshadowed.</p> <p>Already in 2026, <a href="https://www.techtarget.com/ai/news/366644478/SpaceX-IPO-aims-for-AI-and-orbital-data-centers">SpaceX</a> pulled off the largest IPO in history when it raised $75 billion. Just as Databricks was announcing its latest funding round, reports surfaced that <a href="https://www.techtarget.com/ai/news/366624572/Anthropic-intros-next-generation-of-Claude-AI-models">Anthropic</a> may be planning an IPO in October that could surpass SpaceX's. In addition, <a href="https://www.techtarget.com/ai/news/366628814/OpenAIs-GPT-5-is-out-Where-it-shines-and-where-it-doesnt">OpenAI</a> has already filed initial paperwork for an IPO.</p> <p>Those IPOs could take both potential investors and the infrastructure required to go public -- the investment firms that manage such undertakings -- away from Databricks.&nbsp;</p> <p>"Ghodsi has been open that he views this as a poor year to go public with SpaceX, OpenAI, and Anthropic all crowding the IPO conversation, and he's pointed to a possible window in 2027," Mohan said.</p> <p>One thing Databricks does not have to worry about is missing its opportunity to go public, which happened to numerous data management and analytics providers in 2022.</p> <p>Vendors including Qlik, <a href="https://www.techtarget.com/data-technologies/news/252477178/ThoughtSpot-IPO-could-be-coming-after-vendor-adds-first-CFO">ThoughtSpot</a> and <a href="https://www.techtarget.com/data-technologies/news/252516835/Pyramid-Analytics-raises-120M-IPO-could-be-next">Pyramid Analytics</a> spoke openly about IPOs, with Qlik even <a href="https://www.techtarget.com/data-technologies/news/252511695/Qlik-planning-an-IPO-files-application-with-the-SEC">filing its initial paperwork</a>. But after tech stocks plummeted in the spring of 2022, the technology landscape changed toward the end of the year, with AI becoming the dominant focus. Since then, niche data management and analytics vendors have had to reposition themselves and have largely lost the favor of the investment community.</p> <p>Qlik, ThoughtSpot and Pyramid remain private.&nbsp;</p> <p>"Databricks is delaying because public markets are already giving it public company resources without public company constraints," Ni said. "Databricks is using the financial flexibility to drive growth rather than prove quarterly predictability. An IPO will become more necessary when the need for employee liquidity, acquisition currency and public market permanence outweighs the benefits of flexibility."</p> </section> <section class="section main-article-chapter" data-menu-title="The outlook"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The outlook</h2> <p>While an IPO is all but inevitable, and before going public Databricks remains able to attract large amounts of funding to fuel seemingly any growth initiative, technology needs to remain the vendor's main priority, according to McKnight.</p> <p>"My focus, and my clients' focus, is less on the financial headlines&nbsp;and much more on the core architectural and product execution&nbsp;they deliver to the enterprise," he said.</p> <p>For example, the <a href="https://www.techtarget.com/enterprise-software/feature/AI-feature-spend-is-the-new-software-cost-control-problem">high cost of AI development</a> is an ongoing concern for many enterprises, in some cases preventing companies from building agents and other applications. In addition, navigating Databricks to develop data and AI products can be complicated. Meanwhile, after acquiring a spate of companies -- including its purchase of PostgreSQL database innovator <a href="https://www.techtarget.com/data-technologies/news/366649200/Databricks-Electric-acquisition-adds-embeddable-PostgreSQL">ElectricSQL</a> this week -- there are still tools that need to be integrated with Databricks' existing platform before customers can take full advantage.</p> <p>"They&nbsp;could&nbsp;neutralize cost opacity, deliver a true zero-administration experience,&nbsp;provide prebuilt AI templates within Agent Bricks&nbsp;and see about&nbsp;integrating [acquisitions such as] PostgreSQL Lakebase (Neon)&nbsp;and Tabular under Unity Catalog," McKnight said, noting that some competitors deliver faster execution and lower costs across customers data estates.</p> <p>"Additionally, Databricks suffers from severe competitive multi-year ingestion pipeline degradation&nbsp;and critical autoscaling endpoint failures under real-time concurrent inference," he added.</p> <p>Mohan similarly suggested that while a well-timed IPO rather than another funding round is essential, so is ensuring that the many new features Databricks has <a href="https://www.techtarget.com/data-technologies/news/366637142/New-Databricks-tool-aims-to-up-agentic-AI-response-accuracy">developed</a> and <a href="https://www.techtarget.com/data-technologies/news/366588032/Databricks-1B-plus-Tabular-acquisition-adds-Iceberg-support">acquired</a> work as intended, and do so in an integrated manner that enables customers to build the data and AI tools they desire.</p> <p>"Proof that their expansion is actually working," he said when asked what he'd like to see from Databricks over then next year. "There is no guarantee that their forays into marketing and security -- or even Lakebase -- will appeal to buyers who may be reluctant to put all their eggs in one basket. I'd like to see them prove these expansions merge into one coherent product rather than a widening surface."</p> <p><i>Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.</i></p> </section> Experts note that the benefits of going public ultimately outweigh those of staying private, even with the vendor seemingly able to raise unlimited financing. https://cdn.ttgtmedia.com/rms/onlineimages/money_g1222040206.jpg https://www.techtarget.com/data-technologies/news/366649434/IPO-inevitable-for-Databricks-after-adding-5B-in-funding Fri, 14 Aug 2026 14:43:00 GMT <p>While every enterprise software category is adjusting to the disruption caused by AI, the impact has been particularly significant in the BI and analytics sector. That is creating challenges for existing BI vendors -- and potential opportunities for AI-native analytics startups.</p> <p>A key barrier to effective analytics has always been translating business questions and intent into SQL code, data models and data visualizations -- and then translating the analysis results into useful insights. As far back as SQL Server 2000, Microsoft offered a natural language interface for querying databases through its short-lived English Query technology. Efforts to simplify BI for users gained momentum over the next decade with the emergence of <a href="https://www.techtarget.com/data-technologies/tip/9-best-practices-for-self-service-analytics">self-service BI tools</a> offering visual query builders and drag-and-drop interfaces. Those features became defining components of BI software.</p> <p>Almost overnight, generative AI (GenAI) <a href="https://www.techtarget.com/data-technologies/feature/AI-in-business-intelligence-How-to-manage-it-effectively">upended the BI process</a> by providing natural language features that any business user can adopt, along with an unprecedented capability to work with complex datasets. The <a href="https://www.techtarget.com/data-technologies/feature/How-agentic-AI-amplifies-data-management-challenges">rise of agentic AI</a> has further transformed BI environments by enabling organizations to deploy agents that can autonomously analyze data and act on the results.</p> <p>BI vendors responded to these technology changes by adding various AI-driven features: conversational interfaces, augmented data preparation, automated insights and, more recently, agentic analytics capabilities. They hoped doing so would offer existing users the benefits of a streamlined, more insightful UX built on an established platform already trusted by BI, data and IT leaders. But many traditional BI vendors are not having an easy time of it from a business standpoint, and new competitors are targeting their users with AI-native analytics software unencumbered by legacy technology.</p> <section class="section main-article-chapter" data-menu-title="AI-driven challenges -- and more challengers"> <h2 class="section-title"><i class="icon" data-icon="1"></i>AI-driven challenges -- and more challengers</h2> <p>Earlier this year, both Tableau and Qlik <a href="https://www.techtarget.com/data-technologies/news/366645161/Tableau-Qlik-in-flux-What-it-means-for-BI-users-amid-AI-shift">named new top executives</a> and shifted their BI strategies toward agentic analytics. In July, Domo <a href="https://www.techtarget.com/data-technologies/news/366646124/Domo-acquired-for-400M-sells-to-Progress-Software">agreed to be acquired</a> by Progress Software for $400 million, less than 20% of its peak valuation. ThoughtSpot has seen GenAI erode its key differentiator -- a natural language search capability built into its BI software from the start; it now positions the software <a href="https://www.techtarget.com/data-technologies/news/366639258/ThoughtSpot-boosts-agentic-push-with-Analyst-Studio-update">as an agentic analytics platform</a>.</p> <p>While the challenges some BI vendors face stem in part from internal issues, they are all now contending with the same core problem: keeping current customers satisfied while <a href="https://www.techtarget.com/data-technologies/news/366642778/Tableau-repositions-for-AI-unveils-new-knowledge-layer">pivoting toward AI technologies</a> that have radically changed the BI process.</p> <p>To add to their troubles, two AI-native analytics startups, both founded by experienced BI leaders, are gaining increased attention from venture capital investors and early users:</p> <ul class="default-list"> <li>Golden Analytics emerged from stealth in April and <a target="_blank" href="https://www.prnewswire.com/news-releases/golden-analytics-secures-14-million-seed-extension-302794949.html" rel="noopener">launched</a> a public beta of its AI-native analytics platform in June. Led by CEO Francois Ajenstat, former chief product officer at Tableau and analytics vendor Amplitude, Golden has raised $21 million in two rounds of seed funding this year.</li> <li>Gravity launched Orion, an autonomous AI analyst tool, in September 2025 and <a target="_blank" href="https://www.bygravity.com/blog/seed-round" rel="noopener">announced</a> a second funding round in April, taking its total funding to $10 million. It was co-founded by CEO Lucas Thelosen and CTO Drew Gillson, former product leaders at Google and Looker, a BI vendor Google bought in 2020.</li> </ul> <p>These startups aim to inherit segments of the user communities created in the self-service BI era, without being weighed down by the renewal revenue that obliges incumbent vendors to preserve existing workflows for their installed bases. Golden and Gravity are freer to innovate as a result, which could help build their user bases.</p> </section> <section class="section main-article-chapter" data-menu-title="Golden's aim: More effective analytics through AI"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Golden's aim: More effective analytics through AI</h2> <p>In a briefing I had with Ajenstat recently, he emphasized Golden's freedom from legacy technology and its embrace of GenAI. Multiple large language models are at the core of its platform, whereas established BI vendors are attaching LLM technology to products that predate it.</p> <p>Golden generates data visualizations and text-based analytical narratives in response to user questions. Through a "slider of autonomy" control, individual users can decide how much the software does on its own. This reflects Golden's focus on enabling data analysts to <a href="https://www.techtarget.com/data-technologies/feature/How-to-get-reliable-BI-insights-from-AI-augmented-analytics">work more efficiently and productively</a>. That marketing message is directly targeted at Tableau users -- particularly the many who feel underserved by the BI vendor under parent company Salesforce, which acquired Tableau in 2019.</p> <p>For data governance, Golden relies on transparency and human control. For example, any chart exposes the SQL code that queries the cloud data warehouse the software is connected to. Typically, that's Snowflake, but Golden also supports other data platforms, such as Databricks, Google BigQuery and Amazon Redshift.</p> </section> <section class="section main-article-chapter" data-menu-title="Gravity's Orion: A virtual analytics co-worker"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Gravity's Orion: A virtual analytics co-worker</h2> <p>In contrast to Golden's focus on accelerating analytics work by individual BI users, Gravity's Orion software is an agentic analytics system for the enterprise. In a briefing, Thelosen told me that Orion is built to act as a virtual co-worker of data analysts. It uses multiple agents to handle both recurring data analysis tasks and heavier-duty analytics work, such as ad hoc deep dives, root cause analysis and anomaly detection. Data teams act as "conductors" who direct the system's work, Thelosen said.</p> <p>But instead of answering questions framed and posed by an analyst, Orion investigates data unprompted. For example, it might detect a drop in revenue, trace the issue to one country and a single day, and then assess whether that's a genuine insight or an anomaly due to a <a href="https://www.techtarget.com/data-technologies/feature/8-proactive-steps-to-build-trusted-data-for-analytics-and-AI">data quality problem</a>. Based on its findings, Orion delivers tailored reports, slide decks and dashboards to data analysts or business users.</p> <p>Supported data sources include BigQuery, Looker, Snowflake, Databricks, PostgreSQL and other platforms. Gravity designed Orion to be cautious about its own discoveries: built-in quality assurance agents interrogate every analytics result and document the sources and the steps the system takes so that data teams using the software can review its findings.</p> </section> <section class="section main-article-chapter" data-menu-title="AI-native analytics and the data analyst"> <h2 class="section-title"><i class="icon" data-icon="1"></i>AI-native analytics and the data analyst</h2> <p>Large organizations are already trying out these newcomers. Ajenstat said 20% of Golden's nearly 1,000 early-access requests came from the Fortune 500, and Thelosen said Gravity is making strong inroads with publicly traded companies.</p> <p>But such customers are unlikely to fully migrate their analytics environments to AI-native platforms. Just as there is still a market for data warehouses after several decades, there will still be a market for existing BI tools that deliver <a href="https://www.techtarget.com/data-technologies/tip/Why-a-dashboard-audit-matters-before-BI-cleanup">repeatable dashboards and reports</a> used to track KPIs and strategic indicators.</p> <p>Nor are autonomous AI tools likely to completely <a href="https://www.techtarget.com/data-technologies/opinion/Will-AI-replace-data-analysts-A-year-and-a-half-later">take over higher-level analytics functions</a> anytime soon. Golden recognizes this by supporting fully manual data exploration at one end of its slider of autonomy. An FAQ section on Gravity's homepage includes a telling question: <i>Does Orion replace my data team?</i> The answer is plain enough: <i>No. Orion handles the routine analysis that consumes your data team's time ... so they can focus on strategic work. Think of it as adding a tireless senior analyst to the team, not replacing the team.</i></p> <p>At the same time, actual senior analysts who honed their skills with self-service BI tools are not so tireless. Many are now actively looking to use these new AI-native analytics tools to support the next phase of their working practice. It's accelerating a fundamental shift for BI, one that's comparable to the transformation at the start of the self-service era -- and could become even more disruptive.</p> <p><em>Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> Traditional BI vendors face challenges as they try to transition into the AI era -- and now they're being targeted by AI-native startups unencumbered by legacy software. https://cdn.ttgtmedia.com/visuals/searchBusinessAnalytics/data_analytics/businessanalytics_article_013.jpg https://www.techtarget.com/data-technologies/news/366649201/What-AI-native-analytics-startups-offer-that-BI-vendors-dont Thu, 13 Aug 2026 20:47:00 GMT <p>Databricks is going Electric.</p> <p>Though not as historic as Bob Dylan's move away from the acoustic guitar at the 1965 Newport Folk Festival, Databricks on Aug. 11 revealed the acquisition of startup ElectricSQL to add new <a href="https://www.theserverside.com/tip/MySQL-vs-PostgreSQL-Compare-popular-open-source-databases">PostgreSQL database capabilities</a>.</p> <p>Financial terms of the deal were not disclosed. Electric raised $600,000 in pre-seed funding in 2022.</p> <p>Databricks first added PostgreSQL database capabilities with its May 2025 <a href="https://www.techtarget.com/data-technologies/news/366623864/Databricks-adds-Postgres-database-with-1B-Neon-acquisition">acquisition of Neon</a>, which Databricks has subsequently transformed into the core of <a href="https://www.techtarget.com/data-technologies/news/366638723/Databricks-launches-PostgreSQL-Lakebase-to-aid-AI-developers">its Lakebase platform</a>. The acquisition of Electric allows Databricks to extend PostgreSQL database capabilities beyond data lakehouses to edge devices, including agents, additional AI applications and mobile devices.</p> <p>Perhaps the biggest benefit of the acquisition is that Databricks will enable agents to more quickly and easily fix themselves when they take missteps, given that the data they need to do so will be more readily available than before, according to Mike Leone, an analyst at Moor Insights &amp; Strategy.</p> <p>"[The acquisition] definitely matters, but it's more of a specific fix than a big swing," he told TechTarget, adding that agents need to keep notes as they operate to see when they go wrong and correct any errors. "Electric shrank Postgres down small enough to run beside the agent instead of across a network. That makes it fast, and it lets the agent undo a step cleanly instead of leaving a half-finished mess."</p> <p>Stephen Catanzano, an analyst at Omdia, a division of TechTarget, called Databricks' acquisition of Electric significant. Like Leone, he noted that it addresses a specific need in agentic AI development and deployment.</p> <p>"The acquisition addresses a critical infrastructure gap in the emerging world of agentic applications, where AI agents need both ultra-low latency access to local data in sandboxed environments and real-time synchronization with centralized systems," he told TechTarget. "Electric … extends Databricks' existing Postgres capabilities from the centralized lakehouse to the edge where agents actually execute."</p> <p>San Francisco-based Databricks' purchase of Electric is the latest in a series of acquisitions Databricks has made over the past few years to accelerate its expansion beyond data management <a href="https://www.techtarget.com/data-technologies/news/366633904/New-Databricks-tools-target-successful-agentic-AI-development">into AI development</a>.</p> <section class="section main-article-chapter" data-menu-title="Creating a competitive edge"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Creating a competitive edge</h2> <p>PostgreSQL, an open source format for storing disparate data types, has emerged as a popular means of managing data for agents and other AI applications. By 2024, PostgreSQL had become <a target="_blank" href="https://survey.stackoverflow.co/2024/technology" rel="noopener">the most popular database</a>, ahead of MySQL, Microsoft SQL Server, MongoDB and Redis.</p> <p>Versatility -- storing geospatial, time series, JSON and vector database workloads -- is one reason for PostgreSQL's popularity. A large, vibrant community that contributes to its evolution is another.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> The acquisition addresses a critical infrastructure gap in the emerging world of agentic applications, where AI agents need both ultra-low latency access to local data in sandboxed environments and real-time synchronization with centralized systems. </figure> <figcaption> <strong>Stephen Catanzano</strong>Analyst, Omdia </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>Given PostgreSQL's growing role in the AI pipeline, Databricks, <a href="https://www.techtarget.com/data-technologies/news/366625068/Snowflake-acquisition-of-Crunchy-Data-adds-Postgres-database">Snowflake</a> and <a href="https://www.techtarget.com/data-technologies/news/366633563/Streaming-vendor-Redpanda-buys-SQL-engine-unveils-AI-suite">Redpanda</a> all acquired PostgreSQL database capabilities in 2025. Now, by adding Electric's embeddable PostgreSQL databases, Databricks is extending such capabilities beyond traditional data and AI pipelines. In addition, Electric founders James Arthur and Kyle Matthews are joining Databricks as part of the acquisition.</p> <p>Based in Berkeley, Calif., Electric is the developer of <a href="https://pglite.dev/docs/about">PGlite</a>, a lightweight WebAssembly (WASM) version of a PostgreSQL database small enough to be embedded into agents and other applications. With PGlite embedded, applications can <a href="https://www.techtarget.com/data-technologies/news/366646252/Context-is-king-as-agents-evolve-but-problems-persist">access appropriate context</a> where they run rather than having to call back to traditional databases and other data repositories.</p> <p>In addition, Electric provides a real-time sync engine that connects data between agents and a centralized PostgreSQL architecture so that agents can access an organization's data of record in the cloud, as well as the local context that enables them to deliver outputs based on relevant data.</p> <p>Once Electric's capabilities are integrated with Lakebase, Databricks users will be able to deploy swarms of agents that stay current while working alongside one another.</p> <p>"Lakebase already made Postgres fast for agents," Devin Pratt, an analyst at IDC, told TechTarget. "Electric moves it the last few inches, right into the sandbox itself."</p> <p>Meanwhile, though Databricks competitors such as AWS, Snowflake and Microsoft provide PostgreSQL databases, the acquisition of Electric adds embeddable PostgreSQL capabilities that <a href="https://www.techtarget.com/data-technologies/news/366617350/Record-funding-round-reflects-Databricks-differentiation">distinguish Databricks</a> from its peers, Pratt continued.</p> <p>"Snowflake bought its Neon, [but] nobody's bought their Electric yet," he said. "That's the gap Databricks just closed on itself."</p> <p>Catanzano similarly noted that adding Electric's capabilities will help Databricks stand apart from competitors. While other providers offer managed PostgreSQL and data synchronization services, they are designed for traditional database workloads rather than the unique requirements of agents that access the data they need at runtime, operate in distributed sandboxed environments, and <a href="https://www.techtarget.com/data-technologies/news/366646162/Potential-consequences-are-severe-when-AI-agents-lack-context">require both individualized context</a> and a shared state.</p> <p>"Electric's combination of embeddable WASM Postgres with real-time sync architecture specifically designed for agent collaboration represents a more specialized solution for this emerging use case, positioning Databricks ahead of the curve … rather than simply keeping pace," Catanzano said.</p> <p>Leone, however, noted that while Electric's capabilities provide some differentiation, they aren't completely unique. For example, startup Turso similarly provides each agent with its own small database.</p> <p>"What's less common is Databricks insisting the small copy next to the agent be the exact same database as the big one in the middle, not a lookalike," he said.</p> </section> <section class="section main-article-chapter" data-menu-title="Additive acquisitions"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Additive acquisitions</h2> <p>While acquiring Electric adds capabilities that help distinguish Databricks among the vendors racing to add tools that help customers build AI tools, it's just the latest in a long line of acquisitions Databricks has made over the past few years to quickly add capabilities that fuel AI development.</p> <p>In addition to Neon and Electric, Databricks acquisitions include <a href="https://www.techtarget.com/data-technologies/news/366542389/Databricks-acquiring-MosaicML-to-add-more-generative-AI">MosaicML</a>, which forms the foundation of the Mosaic AI development platform, along with Arcion, BladeBridge, Einblick, Lilac AI, Mooncake and Tabular to add capabilities that complement Mosaic AI and Lakebase.</p> <p>"They've been incredibly smart, and unusually consistent about it," Leone said. "Databricks buys things it would otherwise spend years building itself, and it buys the people along with the technology, which is why the same names keep showing up. Electric fits that perfectly."</p> <p>In Electric's case, engineering a full database to run in a small workspace such as an agent's <a href="https://www.techtarget.com/cybersecurity/definition/sandbox">sandbox</a> is different than engineering a database to run in the cloud, he continued.</p> <p>"Electric has already solved it," Leone said. "The Electric team is joining the same group that came over with Neon, so that's two database companies now folded into one team on purpose."</p> <p>Pratt likewise noted that Databricks' acquisition strategy has been effective to date. In particular, MosaicML and Neon have become key parts of the vendor's platform, though Tabular's capabilities, which added support for Apache Iceberg storage, have been slower to integrate, he continued.</p> <p>"Databricks isn't buying new products, it's buying pieces that already have a home waiting for them," Pratt said.</p> </section> <section class="section main-article-chapter" data-menu-title="Looking ahead"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Looking ahead</h2> <p>With Databricks <a href="https://www.techtarget.com/data-technologies/news/366636532/Databricks-adds-4B-funding-round-IPO-could-be-next">continuing to raise capital</a> -- the vendor revealed on Thursday that it added $5 billion in venture capital funding that would bring its total funding to over $30 billion -- more acquisitions are possible.</p> <p>Leone, however, advised Databricks to concentrate at least some resources on delivering new capabilities it <a href="https://www.techtarget.com/data-technologies/news/366644813/Databricks-intros-new-Genie-data-management-tools-to-aid-AI">publicly revealed in recent months,</a> but hasn't yet made generally available. In addition, making its growing platform easier to navigate would be wise, Leone continued.</p> <p>"A lot of what Databricks announced this year is still labeled beta or preview, meaning it isn't fully released," he said. "Getting that work done is worth more to current customers than another round of new features. I'd also want them to say plainly which tool to use for which job, because the platform has gotten big enough that its size is now the problem."</p> <p>Pratt, meanwhile, suggested that Databricks continue to ensure that the capabilities it is acquiring get integrated in a manner that makes them both additive as well as easy to use with the rest of <a href="https://www.techtarget.com/data-technologies/news/366639354/New-Databricks-tool-targets-streaming-data-cost-complexity">the vendor's platform</a>.</p> <p>"Neon to Lakebase took about a year -- that's the pace the rest of the portfolio should be held to," he said.</p> <p><i>Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.</i></p> </section> Flush with funding, the data management and AI vendor continues to make purchases that add capabilities aimed at aiding users building and deploying agents. https://cdn.ttgtmedia.com/rms/onlineimages/money_g1250581414.jpg https://www.techtarget.com/data-technologies/news/366649200/Databricks-Electric-acquisition-adds-embeddable-PostgreSQL Thu, 13 Aug 2026 17:15:00 GMT <p>MongoDB on Thursday unveiled new features designed to connect operational data with AI development tools so users can easily and quickly feed agents and other applications the context they need to deliver accurate outputs.</p> <p>Revealed during MongoDB.local Build Fest, a user event in San Francisco, the new capabilities, among others, include connectors and integrations with agentic coding platforms such as Grok Build and Vercel, a managed <a href="https://www.techtarget.com/data-technologies/feature/One-year-of-MCP-Support-a-must-for-data-management-vendors">Model Context Protocol (MCP) server</a> for MongoDB's Atlas database service, and automated Voyage AI <a href="https://www.techtarget.com/data-technologies/feature/Vector-search-now-a-critical-component-of-GenAI-development">vector embeddings</a> in MongoDB Atlas.</p> <p>Collectively, MongoDB's latest set of tools are significant additions for existing users given that they unify capabilities in one platform that otherwise must be pieced together with tools from different vendors, according to Stephen Catanzano, an analyst at Omdia, a division of TechTarget.</p> <p>"This is a strong set of capabilities that collectively solve … the operational complexity and data staleness that comes from stitching together separate systems for databases, vector stores and embeddings," he told TechTarget. "They unify live operational data with AI retrieval and agent workflows, eliminating synchronization overhead while letting developers work inside tools they already use."</p> <p>Unification, in fact, is how MongoDB is distinguishing itself from competing vendors that are also introducing features designed to aid AI development, Catanzano continued.</p> <p>"Competitors typically require teams to bolt together separate operational databases, vector stores and embedding services that need constant synchronization," he said. "The emphasis on real-time data access combined with Voyage AI models and automated embedding management that eliminates custom pipelines, sets it apart from the fragmented architectures most vendors require."</p> <p>Based in New York City, MongoDB is a longtime database specialist that, like competing database vendors such as <a href="https://www.techtarget.com/data-technologies/news/366645317/Couchbase-evolution-continues-with-new-data-layer-for-AI">Couchbase,</a> along with broader data management vendors such as <a href="https://www.techtarget.com/data-technologies/news/366644813/Databricks-intros-new-Genie-data-management-tools-to-aid-AI">Databricks</a> and <a href="https://www.techtarget.com/data-technologies/news/366643795/Snowflake-barrage-adds-more-AI-development-analysis-tools">Snowflake</a>, has expanded into AI development over the past few years.</p> <section class="section main-article-chapter" data-menu-title="Aiding AI development"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Aiding AI development</h2> <p>Despite the high priority data management and AI providers have placed on simplifying AI development, building agents and other AI tools that perform as intended in production remains challenging. Heading into 2026, <a target="_blank" href="https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html?id=us:2ps:3gl:aisgm26:awa:CONS:em:K0218784:012626:kwd-430833501819:195648817121:794247818306::&amp;gclsrc=aw.ds&amp;gad_source=1&amp;gad_campaignid=23269751971&amp;gclid=CjwKCAjws_DTBhB_EiwAXZknGeJC8uqNU9YNaTWHUXJEhY_qQNxuAtQrd6VPywO0mCMZiX_C4vmHshoCupUQAvD_BwE" rel="noopener">Deloitte reported</a> that only one-quarter of all enterprises expected even 40% of their AI projects to ever make it past the pilot stage.</p> <p>In response, vendors ranging from tech giants such as AWS and Microsoft to specialists including Alteryx and Tableau have all introduced features designed to enable customers to more successfully build AI tools. In particular, developing tools that enable enterprises to build a data layer that <a href="https://www.techtarget.com/data-technologies/news/366646252/Context-is-king-as-agents-evolve-but-problems-persist">feeds AI tools relevant context</a> has been a focal point for many vendors.</p> <p>That includes MongoDB.</p> <p>In January, the vendor <a href="https://www.techtarget.com/data-technologies/news/366637414/MongoDB-launches-latest-Voyage-models-to-aid-AI-development">launched new Voyage AI models</a> to improve data retrieval for agents, in May MongoDB <a href="https://www.techtarget.com/data-technologies/news/366642768/MongoDB-adds-new-vector-performance-capabilities-to-aid-AI">released new vector indexing capabilities</a> and in June <a href="https://www.techtarget.com/data-technologies/news/366645316/Latest-MongoDB-tools-tackle-top-AI-development-hurdles">introduced tools</a> that further address feeding agents with relevant context.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> This is a strong set of capabilities that collectively solve … the operational complexity and data staleness that comes from stitching together separate systems for databases, vector stores and embeddings. </figure> <figcaption> <strong>Stephen Catanzano</strong>Analyst, Omdia </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>Now, MongoDB is continuing its efforts to aid AI development more capabilities&nbsp;</p> <p>The addition of the specific capabilities unveiled on Wednesday was motivated by a mix of customer feedback and MongoDB's own observations of how the AI development landscape is evolving, according to Ben Cefalo, the vendor's chief product officer.</p> <p>"The tools people build will change quickly, and if MongoDB isn't already present wherever that work is happening, [we would be] asking builders to step outside their workflow to use us," he said. "That's the wrong direction. … For us, it's the same principle MongoDB was founded on, [which is to] meet developers on their own terms, wherever they are."</p> <p>The new features with the most potential value for users are the managed MCP server for Atlas and Automated Voyage AI vector embeddings in MongoDB Atlas, according to Catanzano.</p> <p>The first provides developers with a managed version of <a href="https://www.techtarget.com/data-technologies/news/366631455/MongoDB-adds-MCP-server-expands-AI-development-capabilities">the vendor's MCP server</a>, which was first launched in September 2025 and enables users to standardize connections between agents and MongoDB. The second automates &nbsp;the generation of vectors that make data discoverable to agents.</p> <p>"The managed MCP server provides a zero-maintenance, standards-based way for any agent to access MongoDB data with proper governance controls, while automated embeddings solve the painful synchronization problem by generating and updating embeddings inside the database as documents change," Catanzano said.</p> <p>In addition to the managed MCP server for Atlas and Automated Voyage AI vector embeddings in MongoDB Atlas, MongoDB's new capabilities include the following:</p> <ul type="disc" class="default-list"> <li>Integrations with agent-powered coding platforms Cursor, Devin, Grok Build and Vercel, including a native integration between MongoDB Atlas and <a href="https://www.nocode.mba/articles/v0-review-ai-apps">v0 by Vercel</a>.</li> <li>Native connectors between real-time operational data and applications built with Claude and ChatGPT.</li> <li>Atlas App Connections, an identity and access layer that enables users to track and audit an agent's actions.</li> <li>An API that makes MongoDB's embedding and reranking models available through Atlas and accessible to any application whether inside MongoDB's environment or not.</li> <li>A new AI model that enables coding agents to retrieve information from <a href="https://www.techtarget.com/whatis/definition/codebase-code-base">codebases</a> with higher accuracy and lower cost than when doing so from general-purpose embedding models.</li> <li>Vector search in Atlas Stream Processing.</li> </ul> <p>Most of the new capabilities are not yet generally available, although Automated Voyage AI vector embeddings in MongoDB Atlas, the Atlas Embedding and Reranking API, and vector search in Atlas Stream Processing are GA.</p> <p>"This is a strong new set of capabilities that strengthens MongoDB's position as a leading multi-modal database that supports agentic AI," Kevin Petrie, an analyst at BARC U.S., told TechTarget. "MongoDB correctly recognizes that AI projects require diverse structured and unstructured data inputs, along with diverse search and retrieval methods. Its combination of [capabilities] address these needs."</p> <p>Like Catanzano, Petrie noted the value of the managed MCP server for Atlas. In addition, he highlighted the significance of the connectors for users.</p> <p>"The native LLM connectors and MCP server help standardize integration with the rich ecosystem of AI elements," he said.</p> </section> <section class="section main-article-chapter" data-menu-title="Looking ahead"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Looking ahead</h2> <p>As MongoDB plots product development over the final months of 2026, the vendor plans to continue focusing on better enabling the developers and startup companies creating <a href="https://www.techtarget.com/ai/feature/11-real-world-agentic-AI-examples-and-use-cases">innovative AI applications</a> to build new tools, according to Cefalo.</p> <p>"That's a big part of why you're seeing us show up so heavily in San Francisco," he said.</p> <p>In addition, MongoDB aims to increase awareness of its Voyage AI models given how they can enable organizations to run elements of the <a href="https://www.techtarget.com/ai/tip/Agentic-AI-workflows-Trends-examples-and-best-practices">agentic AI workflow</a>, such as vector indexing and data retrieval, without having to piece together tools from various vendors, Cefalo continued.</p> <p>"The direction is consistent: fewer systems, better accuracy and less babysitting infrastructure," he said.</p> <p>Catanzano, meanwhile, suggested that MongoDB add more agent governance capabilities as more organizations begin to move agents into production environments. In addition, he noted that prebuilt agent templates for common uses such as customer support and <a href="https://www.techtarget.com/ai/opinion/How-to-manage-the-gap-between-enterprise-AI-use-and-AI-regulation">compliance monitoring</a> could aid customers that lack the expertise to build such tools from scratch.</p> <p>"MongoDB could deepen agent governance with comprehensive observability and policy enforcement for production deployments, build cost optimization features that intelligently route between models based on accuracy and budget tradeoffs, and offer prebuilt agent templates for common enterprise use cases," he said.</p> <p><i>Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.</i></p> </section> Unifying frequently disparate capabilities such as vector embedding, data retrieval and agent workflows helps distinguish the vendor amid a crowded field of data and AI providers. https://cdn.ttgtmedia.com/rms/onlineimages/machine%20learning_g1186820873.jpg https://www.techtarget.com/data-technologies/news/366649199/MongoDB-intros-latest-features-to-fuel-AI-development Thu, 13 Aug 2026 17:03:00 GMT <p>Data mesh promises faster access to trusted data, but only if organizations are ready and willing to rethink who owns it.</p> <p>As demands for AI and analytics grow, enterprise data teams face increased pressure. Data mesh is a decentralized alternative to traditional centralized data management by shifting responsibility to the business domains most familiar with the data. This approach can enhance data quality, governance and accessibility, but adopting it can be a heavy lift, according to experts.</p> <p>Data mesh is fundamentally a paradigm shift that requires new data management techniques – and for leaders to assess their teams current data skills and cultural readiness to manage data as a product. While that organizational change makes data mesh appealing, executing it can be difficult.</p> <p>"If you're looking to really accelerate your AI with good, clean data that's more trustworthy and more discoverable, data mesh is the way to go. But it's not something you can flip the switch and go to all at one time," said Matt McClelland, senior managing director of FTI Consulting.</p> <section class="section main-article-chapter" data-menu-title="Why data mesh appeals to data leaders"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why data mesh appeals to data leaders</h2> <p>Traditional centralized data architectures gather data from disparate sources, such as spreadsheets and software applications, and then collect, manage and store it in a unified repository, such as a <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-lake-vs-data-warehouse-Key-differences-explained">data warehouse or data lake</a>. But now, organizations are turning to decentralized data architectures, including data mesh, because centralized models can overload data teams with dashboard requests, pipeline changes and other data-related needs from across the entire business.<br><br>Introduced in 2019, data mesh is a "socio-technical" approach that embraces a "divide and conquer" mentality. Data mesh distributes data ownership across business domains rather than concentrating responsibility within a central data team. In data mesh, domain teams manage and are accountable for their data, said Noel Yuhanna, vice president and principal analyst at Forrester.</p> <p>According to Yuhanna and other data leaders, placing accountability with domain teams that best understand the data and its regulations, restrictions and governance needs makes sense. This approach can yield higher-quality, more accessible data without sacrificing security, privacy and compliance, they said.</p> <p>"That's a nirvana state that people want to get to," Yuhanna said. "But getting to that state is not trivial."</p> </section> <section class="section main-article-chapter" data-menu-title="Why data mesh adoptions stall"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why data mesh adoptions stall</h2> <p>Technology supports data mesh, but no single platform or tool can deliver the operating model on its own. Yuhanna described the move to data mesh as a socio-technical approach that requires changes to processes, culture and people.</p> <p>"[Data mesh] requires people to think differently," Yuhanna said.</p> <p>This change requires <a href="https://www.techtarget.com/ai/tip/How-to-use-change-management-for-better-AI-adoption">strong change management</a> to help teams adopt new responsibilities and workflows. &nbsp;</p> <p>Executives should anticipate resistance to the new approach, he added, noting that domain teams sometimes don't want to share data or collaborate with other domains or IT. Executives also might resist moving to data mesh after years of investment in centralized data management approaches, experts said.</p> <p>An organization's existing data ecosystem can also be a barrier to data mesh adoption, with legacy technology, existing data silos and complexity often stymying efforts, Yuhanna said. Some organizations get only a fraction of their data -- as low as 1% -- into a data mesh, which limits the value, he added.</p> <p>Conventional approaches to funding data management can further frustrate attempts to implement data mesh. Because this model involves shared work across domains, organizations must find ways to fund priorities that span multiple departments.</p> <p>"If you have a centralized funding model and not cross-functional funding, that can be a challenge," McClelland said.</p> <p>Many domain teams lack the necessary data engineering, product management and governance skills to own data products used in a data mesh model. McClelland said data mesh forces less technical domain teams, such as marketing or HR, to operate like software product teams. <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Data-literacy-training-requires-a-dual-approach">Training in this area</a> can address that gap.</p> <p>"Most domain teams lack data engineering expertise, and forcing them to manage data pipelines without proper training leads to broken infrastructure and poor data quality," McClelland said.</p> <p>Some organizations still fall short on data management maturity, which is necessary for a effective data mesh adoption. Automated data classification, security controls, compliance checks and <a href="https://www.techtarget.com/data-technologies/tip/9-metadata-management-standards-that-guide-success">metadata standards</a> should be built into the platform from the start to avoid an unmanageable "data swamp," McClelland said.</p> <p>He stressed that data mesh doesn't eliminate <a href="https://www.techtarget.com/searchdatabackup/tip/Enterprise-data-governance-Frameworks-and-best-practices">enterprise data governance</a>, noting that an overarching governance framework sits above it.</p> </section> <section class="section main-article-chapter" data-menu-title="Assessing readiness"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Assessing readiness</h2> <p>Data mesh can deliver benefits, but ROI depends on the organization's starting point, Yuhanna said.</p> <p>A distributed data approach needs clear management structures and accountability, otherwise data mesh is unlikely to work, he said. Yuhanna advised executives <a href="https://www.techtarget.com/it-strategy/tip/5-tips-for-creating-a-data-driven-culture">to assess their data culture</a> and evaluate their organizational structure and maturity around data accountability, security and governance. A higher level of maturity makes moving to data mesh less taxing.</p> <p>Organizations must also determine how well IT and business teams collaborate, as well as the level of communication among CDOs, CIOs, engineers, business analysts, data stewards and business users, Yuhanna said. A more collaborative environment makes implementation easier.</p> <p>McClelland said leadership should also assess their maturity around data-related processes, using the following tests:</p> <ul class="default-list"> <li><b>Time-to-data audit.</b> McClelland advised businesses to measure how long it takes a business unit to request, get approval for and ingest a new external data source. "If the bottleneck is purely bureaucratic rather than technical, the culture is highly resistant to self-serve models," he said.</li> <li><b>Domain capability mapping.</b> This test audits the <a href="https://www.techtarget.com/searchcio/tip/Digital-literacy-vs-digital-fluency-Learn-the-differences">technical literacy</a> of business units by classifying domain teams into tiers based on tech capability, McClelland said. "If no business units have the skills to manage their own data endpoints, a mesh is premature," he said.</li> <li><b>"Who pays?" test.</b> "Propose a scenario where Domain A needs a feature built by Domain B's data team," McClelland said. "If your current corporate structure has no mechanism for cross-domain funding or shared priority alignment, the operational model will stall."</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="How leaders can prepare for data mesh"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How leaders can prepare for data mesh</h2> <p>The scope and complexity of the challenges aren't insurmountable, especially if executives keep some best practices in mind as they move to a data mesh architecture, experts said.</p> <h3>Pre-implementation</h3> <p>First, stakeholders must formalize a plan. Ad hoc adoption is not an option, Yuhanna said, and emphasized the need for defined policies, standards and requirements.</p> <p>During planning, organizations should <a href="https://www.techtarget.com/data-technologies/tip/How-to-develop-a-data-governance-strategy-7-key-steps">embed security and governance from the start</a>, Yuhanna added. But McClelland warned against relying on documentation or human committees to enforce governance. Instead, <a href="https://www.techtarget.com/searchdatamanagement/tip/How-to-use-AI-to-enforce-data-governance-policies">automated governance</a> is preferable.</p> <p>"Bake compliance, masking rules and access control directly into the self-serve platform deployment templates so domains get compliance 'for free,'" McClelland said.</p> <h3>Implementation</h3> <p>Start with a high-value use case that spans several domains, recognizing the shift is evolutionary, not a big-bang moment, said Bhath Thota, a partner at consulting firm Kearney. Begin with domains with more mature data management practices and staff capable of championing the shift, he added.</p> <p>Alternatively, start by moving small datasets to data mesh, Yuhanna said.</p> <p>Either way, organizations should fund the platform as a product with domain developers treated as customers, McClelland said.</p> <p>"If the self-serve platform is difficult to use, domains will bypass it and build their own rogue infrastructure," he added.</p> <p>Treat the move to data mesh as a culture shift. That shift requires organizations to <a href="https://www.techtarget.com/searchbusinessanalytics/feature/How-business-leaders-can-make-a-data-literate-culture-stick">replace data hoarding habits</a> with an open marketplace that ties data quality to business performance, McClelland said. Technology should support that behavioral shift, he added.</p> <p>"Treating data mesh as a technology migration is the number one reason implementations fail," McClelland said. "Data mesh is inherently a sociotechnical shift."</p> <p>Embedding data product owners into business units will help. Instead of reporting to central IT, employees report to the business domain lead to help align data initiatives with business strategy, McClelland said.</p> <p>And don't aim for perfection, Thota said. Data mesh doesn't have to be all-or-nothing transformation. Many organizations use a hybrid approach that combines data mesh principles with data lakes and data lakehouses.</p> <p>"I have yet to see any company with 100% in data mesh," Yuhanna said.</p> <p><em>Mary K. Pratt is an award-winning freelance journalist specializing in enterprise IT, cybersecurity strategy and data management.</em></p> </section> The technological aspect is only part of the move toward decentralized data management. Adoption can stall when teams resist new responsibilities and cross-domain ways of working. https://cdn.ttgtmedia.com/rms/onlineimages/code_g1255337870.jpg https://www.techtarget.com/data-technologies/feature/Benefits-of-data-mesh-might-not-be-worth-the-cost Wed, 12 Aug 2026 00:00:00 GMT <p>For most of the last 100-plus years, competitive advantage came from three familiar levers: improving operational efficiency, automating repetitive work and reducing labor costs. AI is quietly retiring that playbook and replacing it with one built around optimized decision-making.</p> <p>But the organizations pulling ahead are not the ones with the most AI tools. They're the ones rewriting the economics underpinning their AI-driven business decisions.</p> <p>AI is not just changing the economics of labor. It is also changing the economics of learning. Automation makes today's work less expensive. Learning makes tomorrow's decisions better. Those are different economic engines, and only one of them compounds business value.</p> <p>For data and analytics leaders, that distinction is the whole job. The key mandate is no longer to <a href="https://www.techtarget.com/searchbusinessanalytics/opinion/Why-the-rush-to-replace-dashboards-with-AI-is-a-mistake">deliver useful dashboards</a> and govern data tables for accuracy. Now, it is to build the economic infrastructure of continuous learning and adaptation: the <a href="https://www.techtarget.com/searchdatamanagement/feature/Proactive-practices-for-data-quality-improvement">trusted data products</a>, semantic layers and certified measures that help their organizations to learn faster than rivals and reuse what they learn at near-zero cost.</p> <section class="section main-article-chapter" data-menu-title="From optimizing processes to optimizing decisions"> <h2 class="section-title"><i class="icon" data-icon="1"></i>From optimizing processes to optimizing decisions</h2> <p>The industrial-era analytics stack was built to optimize processes: run the plant hotter, route the truck smarter and close the books faster. Those improvements are valuable, but bounded: over time, process optimization runs headlong into the law of diminishing returns.</p> <p>In the AI era, the opportunity is different. It is about <a href="https://www.techtarget.com/searchenterpriseai/feature/6-key-benefits-of-AI-for-business">improving decisions</a>. Every customer interaction, pricing call and service ticket is a series of choices that can be made more smartly the next time. Treat each of those decisions as a chance to learn, and the organization stops competing on how fast it executes and starts competing on how well it decides. Faster processes have a ceiling. Better decisions do not.</p> </section> <section class="section main-article-chapter" data-menu-title="The economics of reuse"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The economics of reuse</h2> <p>The next lever is reuse. Unlike physical assets, data does not deplete when used. It can serve many use cases at near-zero marginal cost and appreciates in value as it is <a href="https://www.techtarget.com/searchdatamanagement/tip/Assemble-the-layers-of-big-data-stack-architecture">combined with other data and reused.</a> I call the organizational instinct behind this the Marginal Propensity to Reuse (MPR): how broadly a data asset, model or analytical insight gets applied across the enterprise once it's created.</p> <p>This is where trusted data products become appreciating assets rather than depreciating projects. A project is built once, for one purpose, and starts losing value the day it goes into production. A data product is built to be reused. In a 2022 article, McKinsey said treating data like a product can deliver new use cases up to 90% faster and reduce total cost of ownership by 30%. Every reuse drives marginal cost down and cumulative value up.</p> </section> <section class="section main-article-chapter" data-menu-title="The hidden AI tax"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The hidden AI tax</h2> <p>The flip side is economic, too. LLMs do not automatically create reliable organizational knowledge -- they consume it. Point AI at a weakly governed or ungoverned data environment, and it does not fail loudly. It produces confident guesses that don't look wrong, but are.</p> <p>The result is a hidden AI tax: <a href="https://www.techtarget.com/searchenterpriseai/news/366644404/MIT-study-warns-of-major-AI-risk-Is-governance-keeping-up">more hallucinations</a>, more human review, slower adoption, duplicated work and decaying trust in the AI tool. In 2020, McKinsey said data users can spend 30% to 40% of their time searching for data when a clear inventory is not available -- and another 20% to 30% cleaning data when strong controls are not in place. There's no reason to think things have improved since then for organizations with weak data environments.</p> <p>Every point of that tax is business value AI can't realize because the data foundation won't let it.</p> </section> <section class="section main-article-chapter" data-menu-title="The economics of learning"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The economics of learning</h2> <p>Reuse compounds knowledge across the enterprise; learning compounds it over time. The Marginal Propensity to Learn (MPL) is the rate at which an organization improves from each decision, interaction and feedback cycle.</p> <p>Historically, organizational learning crawled; it was locked in individual experience and lost when people left. AI can compress that cycle dramatically, but only when humans design for it by capturing feedback, <a href="https://www.techtarget.com/searchenterpriseai/tip/Exploring-the-context-layer-for-AI-systems">preserving decision rationale</a> and feeding lessons back into the next AI-driven decision.</p> <p>The AI model does not learn on its own. The organization must build the loop -- and then it learns, too, as AI works more effectively.</p> </section> <section class="section main-article-chapter" data-menu-title="Measure value, not activity"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Measure value, not activity</h2> <p>None of this shows up on a traditional analytics scorecard. Dashboard counts, data volume and query performance measure operation, not value. The metrics that matter now -- decision quality, decision velocity, data and knowledge reuse, learning velocity and cross-functional adoption -- are today's unmeasured outcomes. No one built them, so no one governs them.</p> <p>Data teams spent a decade certifying whether metrics are <i>accurate</i>; the next decade belongs to those who can certify <a href="https://www.techtarget.com/searchbusinessanalytics/opinion/Your-AI-isnt-failing-your-metrics-are">whether they are <i>right</i></a>. Standing up those learning and reuse metrics is data work, not AI work.</p> <p>The rise of AI economics is not about replacing people with algorithms. It is about turning trusted data into organizational intelligence that helps companies improve with every decision. Industrial economies rewarded scale; AI economies reward learning and reuse. Companies that improve their decision-making faster through data -- and apply those lessons across the organization -- will build competitive advantages that compound. In an economy powered by learning, the organizations that learn from their data fastest win.</p> <p><i>Bill Schmarzo, "The Dean of Big Data," teaches AI-driven innovation at Iowa State University and advises organizations on data science, AI and data monetization. He is a former executive at Dell Technologies, Hitachi Vantara and Yahoo. He has written books on data-driven innovation and applied AI strategy.</i></p> </section> The companies pulling ahead in AI aren't doing it by deploying more tools. Rather, they're building the data foundations that make intelligence reusable, scalable and reliable. https://cdn.ttgtmedia.com/rms/onlineimages/machine_learning_g1040557296.jpg https://www.techtarget.com/data-technologies/opinion/AI-value-comes-from-continuous-learning-not-automation Fri, 31 Jul 2026 14:40:00 GMT <p>The best AI outputs are fueled by both structured and unstructured data.</p> <p><a href="https://www.techtarget.com/whatis/definition/structured-data">Structured data</a>, by definition, has some kind of order, such as rows and columns. <a href="https://www.techtarget.com/searchbusinessanalytics/definition/unstructured-data">Unstructured data </a>is the opposite, lacking any particular format. The data is just there, without being organized or structured in any particular way.</p> <p>Organizations have both types of data spread across various systems and platforms. The ability to integrate these separate data types is critical as the use of AI technologies continues to grow.</p> <p>This is particularly important for <a href="https://www.techtarget.com/searchenterpriseai/definition/agentic-AI">agentic AI</a>, since an agent plans and executes across multi-step workflows instead of just returning an answer for a person to review.</p> <p>Organizations are not using unstructured data nearly as effectively as they should be. According to IDC, unstructured data accounts for <a target="_blank" href="https://www.ibm.com/think/insights/enable-ai-unstructured-data-integration-governance" rel="noopener">over 90%</a> of enterprise data and is growing three times faster than structured data, but less than 1% is used in generative AI today. As such, many agents are working from a fraction of what an organization stores.</p> <p>To give agents the full picture, IT leadership must ensure that they are getting the most out of the data they have.</p> <section class="section main-article-chapter" data-menu-title="What integration looks like in practice"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What integration looks like in practice</h2> <p>Multiple organizations have successfully combined structured and unstructured data for agentic AI. Two specific public examples can be found in the healthcare and insurance verticals. These examples show what the effort to integrate structured and unstructured data buys: time saved in one industry and money saved in the other.</p> <h3>Healthcare</h3> <p>Healthcare organizations have an overwhelming volume of data to manage. New Jersey-based healthcare provider PromptCare uses an AI agent to review unstructured medical records and assess whether a patient meets therapy qualification criteria.</p> <p>That data is measured against structured qualification criteria for a specific patient before a person makes the final call. According to PromptCare CIO Phil Merrell, the approach cut referral processing time <a href="https://www.techtarget.com/searchcio/feature/Agentic-AI-streamlines-PromptCares-operations">from 15 days to four</a>.</p> <h3>Insurance</h3> <p>Shift Technology's fraud-detection platform reads adjuster and investigator unstructured notes alongside structured claim fields such as policy dates and amounts. A company <u><a target="_blank" href="https://www.shift-technology.com/resources/case-studies/ai-in-action/ai-insurance-advanced-resolution-techniques" rel="noopener">case study</a></u> describes the platform linking a $525,000 kitchen-fire claim to a similar claim from two years earlier, a match that structured data alone would have missed.</p> <div class="youtube-iframe-container"> <iframe id="ytplayer-0" src="https://www.youtube.com/embed/EMY2PBd7wT8?autoplay=0&amp;modestbranding=1&amp;rel=0&amp;widget_referrer=null&amp;enablejsapi=1&amp;origin=https://www.techtarget.com" type="text/html" height="360" width="640" frameborder="0"></iframe> </div> </section> <section class="section main-article-chapter" data-menu-title="How to integrate structured and unstructured data for AI"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How to integrate structured and unstructured data for AI</h2> <p>While the benefit of using both unstructured and structured data is obvious, it's not quite as obvious how to use both types of data together. By definition, unstructured data has no structure, so it won't fit into the databases, tables and rows of structured data.</p> <p>That doesn't mean, however, that there aren't solid best practices for combining the two types of data. IT leaders can structure data pipelines in a way that moves unstructured content out of file shares and email threads and into the same answer an agent already pulls from structured records.</p> <p>Use the following four practices to turn scattered data into a single retrieval process:</p> <ol class="default-list"> <li><b>Consolidate storage in a unified platform.</b> Using platforms such as a <u><a href="https://www.techtarget.com/searchdatamanagement/opinion/Why-agentic-AI-demands-both-structured-and-unstructured-data">data lakehouse</a></u>, which combines a data warehouse's governance and query performance with a data lake's flexibility for unstructured content, can make both data types equally accessible to agentic AI.</li> <li><b>Classify and tag content at ingestion.</b> File names and timestamps are not enough to match unstructured content to the structured records it relates to. Vendor Databricks provides <u><a target="_blank" href="https://docs.databricks.com/aws/en/agents/tutorials/ai-cookbook/quality-data-pipeline-rag" rel="noopener">guidance</a></u> on building unstructured data pipelines, recommending the use of content-based metadata such as topics and named entities. In that approach, an unstructured data document can be linked to the account, claim or patient record it describes.</li> <li><b>Build one retrieval layer instead of two silos.</b> When it comes to actual retrieval, structured and unstructured data are often queried by separate systems. That approach doesn't connect the two data types. Having <a href="https://www.techtarget.com/searchenterpriseai/tip/Advantages-of-a-semantic-layer-for-enterprise-AI">some kind of semantic graph</a>, so that a single query pulls matching records and documents together, is a critical step. A semantic layer turns the tags added in the above step into connections across systems, rather than labels sitting on individual documents.</li> <li><b>Match the query to the data type.</b> Inside that same retrieval layer, precise questions and open-ended questions still call for different handling. A Google study comparing agent retrieval methods found a semantic, metadata-driven agent achieved <a target="_blank" href="https://arxiv.org/abs/2605.28787" rel="noopener">65.7% higher precision</a> retrieving usable data than a baseline agent searching open, unstructured content. That said, the baseline agent answered 40% more questions by casting a wider net. The best approach is to send exact, execution-critical lookups to the structured side and pull only the most relevant handful of unstructured chunks for everything else.</li> </ol> <p><i>Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.</i></p> </section> For AI agents to give the most accurate, informed responses, organizations must provide them with all the facts. Even those that are hidden in an old email or customer claim. https://cdn.ttgtmedia.com/rms/onlineimages/folder-files13.jpg https://www.techtarget.com/data-technologies/tip/Integrate-structured-and-unstructured-data-for-better-AI-outputs Fri, 31 Jul 2026 13:44:00 GMT <p>Data governance often gets a bad rap in organizations: End users see it as an effort to control what they do with data. The data governance team at York University deliberately set out to avoid creating that perception.</p> <p>"<i>Control</i> is not a word we use in data governance at York at all," said Patrick Cernea, director of data strategy and governance at the university in Toronto. "Instead of trying to stop people from doing things, it's more of a way to help them do their jobs better."</p> <p>To be sure, the governance program aims to <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">tighten data processes</a> at York. The program's six core goals include <a href="https://www.techtarget.com/searchdatamanagement/feature/Controlling-data-sprawl-requires-governance-discipline">eliminating data silos and shadow databases</a> and ensuring regulatory compliance through proper data classification and policies governing data collection, use and retention.</p> <p>But governing data and how it's used "is not the endpoint," Cernea said during a webinar hosted by the Data Governance Professionals Organization (DGPO) this month. "We aren't here for data governance's sake. We're here to solve data issues through data governance."</p> <p>That's reflected in the program's four other goals:</p> <ul class="default-list"> <li>Improving data quality and <a href="https://www.techtarget.com/searchdatamanagement/feature/Proactive-practices-for-data-quality-improvement">increasing trust in data</a>.</li> <li>Streamlining secure access to data assets.</li> <li>Improving data literacy.</li> <li>Enhancing data intelligence for better decision-making.</li> </ul> <p>An internal user survey conducted before the governance initiative began in 2020 showed problems in all those areas, Cernea said. Almost 50% of the respondents said relevant data was hard to find in York's systems, and sizable percentages also cited a lack of clear data definitions, access to required data and trust in the data that was available to them.</p> <section class="section main-article-chapter" data-menu-title="Data issues become a pressing priority"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Data issues become a pressing priority</h2> <p>A subsequent review found that, with data ownership decentralized across the university, data quality was inconsistent. Clear accountability for data and guardrails on its management and use were also lacking, according to Cernea. He said addressing the data issues became a priority due to a $2 million digital transformation project and the <a href="https://www.techtarget.com/searchdatamanagement/tip/AI-analytics-push-data-in-use-protection-up-priority-list">growing use of analytics and AI applications</a> at York.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> We aren't here for data governance's sake. We're here to solve data issues through data governance. </figure> <figcaption> <strong>Patrick Cernea</strong>Director of data strategy and governance, <br>York University </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>Now, there are 830 common data definitions in a metadata repository the data governance team created, and 310 data elements have been classified, with more to come as privacy reviews are completed. Data trustees and data stewards have been assigned to all 21 data domains and 45 sub-domains that York prioritized for governance, and 130 governance rules have been formalized across the university.</p> <p>Cernea said a follow-up user survey showed double-digit improvements in views of data findability, definitions and access, and a 6% increase in the share of users who think data is trustworthy. Externally, York won the 2026 DGPO Data Governance Best Practice Award, which recognizes a governance program for its business value and technical excellence.</p> </section> <section class="section main-article-chapter" data-menu-title="A six-year implementation process"> <h2 class="section-title"><i class="icon" data-icon="1"></i>A six-year implementation process</h2> <p>But getting data governance at York to <a target="_blank" href="https://www.yorku.ca/oipa/data-analytics-governance/" rel="noopener">where it is now</a> was a long journey spanning six years, from May 2020 to April 2026.</p> <h3>Years 1-2</h3> <p>Cernea worked with various stakeholders for two years to lay the foundation for the governance program, splitting his time between that and his existing BI strategy role. Key upfront tasks included:</p> <ul class="default-list"> <li>Defining the program's vision and mission.</li> <li>Outlining <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-roles-and-responsibilities-Whats-needed">roles and responsibilities</a>.</li> <li>Setting goals.</li> <li>Creating data governance and data quality playbooks.</li> <li>Identifying data trustees and stewards.</li> </ul> <p>Separate committees were established with the trustees -- senior executives accountable for data in specific domains -- and the stewards, who work at the sub-domain level.</p> <h3>Years 3-4</h3> <p>The program formally took shape in its third and fourth years. During that period, the university promoted Cernea to his current position full-time and hired a chief data officer to oversee the governance program and other data initiatives. It also hired a data governance analyst to create a three-person team that leads the program alongside the two committees.</p> <p>Implementation milestones in Year 3 included developing a data governance charter, a comprehensive <a href="https://www.techtarget.com/searchdatamanagement/tip/Developing-an-enterprise-data-strategy-10-steps-to-take">data and analytics strategy</a>, an institutional data quality framework and associated quality process maps.</p> <p>In Year 4, the team drafted a data governance policy, launched an internal data definitions website and created a data and analytics community of engagement. The CoE was designed to help <a href="https://www.techtarget.com/searchdatamanagement/tip/Use-these-steps-to-successfully-build-your-data-culture">build a stronger data culture</a> at York through quarterly sessions on using data and resolving problems, Cernea said.</p> <h3>Years 5-6</h3> <p>After a multi-stage review process, York's board approved the data governance policy in December 2025. In Years 5-6, the governance team also developed a data literacy course and created multiple governance-related training videos. Other notable steps included documenting data quality rules and standardizing a set of <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more">governance metrics</a> at the institutional level. Operationally, data governance was embedded into the university's planning, budgeting and performance management processes.</p> <p><iframe title="" aria-label="Table" id="datawrapper-chart-3hSzc" src="https://datawrapper.dwcdn.net/3hSzc/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="479" data-external="1"></iframe> <script type="text/javascript">(function(){function e(){window.addEventListener(`message`,function(e){if(e.data[`datawrapper-height`]!==void 0){var t=document.querySelectorAll(`iframe`);for(var n in e.data[`datawrapper-height`])for(var r=0,i;i=t[r];r++)if(i.contentWindow===e.source){var a=e.data[`datawrapper-height`][n]+`px`;i.style.height=a}}})}e()})();</script> </p> </section> <section class="section main-article-chapter" data-menu-title="Technology's role in the governance program"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Technology's role in the governance program</h2> <p>Cernea said the metadata repository that underpins the governance program is built on Data Cookbook, a cloud-based SaaS application. The repository serves as a data catalog, capturing metadata and <a href="https://www.techtarget.com/searchdatamanagement/feature/How-data-lineage-became-a-boardroom-metric">data lineage information</a> from a Databricks-based data lakehouse running in the Azure cloud. It also contains a business glossary, a data dictionary, data classifications and data quality rules.</p> <p>A separate Data Hub repository and portal provides access to public and restricted datasets, along with information about them. The governance team also developed a Data Navigation Assistant chatbot that maps data products by domain and by category -- such as interactive dashboards and static reports -- to help Data Hub users find what they're looking for.</p> <p>While these tools are key components of the data governance program for end users, Cernea said <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">improving data literacy</a>, through both the literacy course and the data and analytics CoE, was especially critical to the program's acceptance.</p> <p>"If you can establish data literacy early on, people are not going to push back as hard when the data governance policy comes," he said. By the time York's policy was approved, he noted, users already understood the importance of effective governance and data quality.</p> <p>The CoE sessions don't focus directly on data governance; Cernea said they're practical meetings on topics such as available dashboards and potential ways to increase student retention based on data analysis. The sessions aren't just presentations -- they include panels, case studies, open discussions and collaborative workgroups. Attendance has reached full capacity at all but one, according to Cernea.</p> <p>"This is how we make data governance fun, for lack of a better word," he said.</p> </section> <section class="section main-article-chapter" data-menu-title="Solving user problems is key to governance success"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Solving user problems is key to governance success</h2> <p>Ultimately, Cernea sees resolving data gaps and problems that hinder users from working effectively as the biggest selling point for a governance program.</p> <p>"Data governance is often seen as slowing things down," Cernea said. "But you can say, 'No, we're doing the opposite. We're enabling people to speed things up and do their jobs better.'" That, he added, "makes data governance more of a capability than a compliance effort."</p> <p>It's a view shared by other forward-looking data governance leaders.</p> <p>Carol Henry, senior director of data governance and analytics at Longo Brothers Fruit Markets, made a similar point to Cernea's in a session during Dataversity's Enterprise Data Governance Online 2026 <a target="_blank" href="https://datagovernanceonline.com/" rel="noopener">conference</a> in January. Longo's, a supermarket chain in Southern Ontario, launched its data governance program in late 2022. Henry, previously the company's HR director, began by interviewing more than 20 executives across all business units about key business decisions, required insights and data challenges.</p> <p>For Henry, understanding business priorities, strategic initiatives, core KPIs and data-related pain points was critical to <a href="https://www.techtarget.com/searchdatamanagement/tip/6-key-steps-to-develop-a-data-governance-strategy">developing a governance strategy</a> that balances more effective data management with increased business agility. "If data governance isn't aligned with that, it won't succeed," she said.</p> <p><em>Craig Stedman is an industry editor at TechTarget who edits and writes articles on data technologies and processes. He has covered enterprise IT for more than 40 years.</em></p> </section> York University has focused its data governance program more on solving data quality, access and trust issues for end users than on controlling their data use. https://cdn.ttgtmedia.com/visuals/searchDataManagement/governance/datamanagement_article_011.jpg https://www.techtarget.com/data-technologies/feature/Yorks-data-governance-program-eyes-capabilities-not-control Thu, 30 Jul 2026 14:11:00 GMT <p>The world has entered the zettabyte era, a period where surging data growth threatens to overwhelm the systems designed to manage it.</p> <p><a target="_blank" href="https://my.idc.com/getdoc.jsp?containerId=US53363625" rel="noopener">IDC reported</a> that global data reached 181 zettabytes in 2025 and is projected to reach 394 zettabytes by 2028, as GenAI and high-value data spur new enterprise data needs. For data leaders, the issue is no longer just how much data can be stored, but whether the combination of infrastructure, automation and governance can work together to keep expanding data estates usable and trustworthy.</p> <p>Organizations that treat data growth only as a capacity problem will struggle, while those that treat storage management as an operational discipline will be better prepared to control costs, reduce risk, and convert data growth into business value.</p> <section class="section main-article-chapter" data-menu-title="Scale changes everything for data operations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Scale changes everything for data operations</h2> <p>The zettabyte era isn't just about more data. It forces organizations to reevaluate how they <a href="https://www.techtarget.com/searchstorage/Data-storage-management-What-is-it-and-why-is-it-important">store, manage, protect</a> and govern large data estates. For example, moving large datasets can become cost-prohibitive, making data residency and placement decisions more important earlier in the architecture process. Likewise, manual administration becomes impractical, and <a href="https://www.techtarget.com/searchdatamanagement/tip/Metadata-management-standards-examples-that-guide-success">metadata becomes essential</a> for finding, governing, and reusing data.</p> <p>Zettabyte-era readiness requires scalable infrastructure, clear governance and a data-driven culture. Organizations that succeed won't necessarily be the ones with the most storage capacity but the ones that build the operational discipline to manage data at this scale.</p> </section> <section class="section main-article-chapter" data-menu-title="Infrastructure readiness"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Infrastructure readiness</h2> <p>Infrastructure readiness means more than just acquiring the capacity to store zettabytes of data but also <a href="https://www.techtarget.com/searchdatacenter/tip/Balancing-automation-with-human-oversight-in-AI-data-centers">designing policy-based automation</a>. Humans simply cannot manage data at this scale. Policy engines, automated data lifecycle management and <a href="https://www.techtarget.com/searchdatamanagement/opinion/Explore-the-benefits-of-AI-for-DataOps">AI-assisted operations</a> help enforce placement, retention and consistent access policies.</p> <p>Additionally, organizations must rethink their performance needs beyond business workloads and consider their management plans. Organizations must consider whether their data and storage management tools have sufficient hardware resources to keep pace with this unprecedented growth.</p> <p>Cost control also becomes more difficult as data volume grows. Small inefficiencies in retention, duplication and placement can lead to significant costs across large estates.</p> <h3>Infrastructure readiness questions</h3> <ul class="default-list"> <li>Can the organization enforce storage, retention and access policies automatically?</li> <li>Can the data be moved, tiered or archived without manual intervention?</li> <li>Can management systems scale with storage capacity, metadata volume and policy complexity?</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Cultural readiness"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Cultural readiness</h2> <p>Cultural readiness means treating data as a shared business asset, not an inventory IT manages.</p> <p>When an organization has a few terabytes of data, manual effort compensates for weak procedures. It's a common scenario where a few employees have extensive institutional knowledge -- someone knows where a file is or which spreadsheet is correct. &nbsp;</p> <p>Organizations must replace these workarounds and other poor data management practices, such as multiple departments storing their own copies of a centralized dataset. That disorganization is tolerable in small environments, but as data and companies grow, it will hinder <a href="https://www.techtarget.com/searchdatamanagement/tip/Experts-share-practices-to-overcome-AI-data-readiness">data readiness efforts</a> by creating duplication, inconsistent definitions and governance gaps.</p> <p>To operate at zettabyte scale, data culture must evolve. Organizations must replace inconsistent procedures and undocumented institutional knowledge with standards for creating, classifying and handling data.</p> <h3>Cultural readiness questions</h3> <ul class="default-list"> <li>Does every critical dataset have an owner?</li> <li>Do employees understand their <a href="https://www.techtarget.com/searchdatamanagement/definition/data-stewardship">data stewardship</a> responsibilities?</li> <li>Are data standards documented, taught and followed consistently?</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Governance readiness"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Governance readiness</h2> <p>Data governance defines the policies, roles and controls for data management. This discipline is critical at scale, because small governance failures affecting a handful of datasets today will compound into bigger issues related to quality and compliance later.</p> <p>Governance structures vary by organization, but they often include naming conventions, classification, lineage and sensitive data tagging. At zettabyte scale, governance is <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-challenges-that-can-sink-data-operations">difficult to sustain</a> without automation. Organizations will need to consider advanced options, such as using AI to automatically catalog data, <a href="https://www.techtarget.com/searchdatamanagement/feature/How-data-lineage-became-a-boardroom-metric">track data lineage</a> and keep data discoverable.</p> <p>Effective governance is also essential for data trust. Without it, poorly documented assets, duplicate datasets and inconsistent definitions diminish business value. Strong governance is far more than security and compliance; it keeps data reliable, discoverable and fit for use.</p> <h3>Governance readiness questions</h3> <ul class="default-list"> <li>Can authorized users locate trusted data quickly?</li> <li>Is <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">lineage visible</a> across data products and systems?</li> <li>Are retention and lifecycle policies enforced automatically?</li> <li>Is data access and usage auditable?</li> </ul> <p><em>Brien Posey is a former 22-time Microsoft MVP and a commercial astronaut candidate. In his more than 30 years in IT, he has served as a lead network engineer for the U.S. Department of Defense and a network administrator for some of the largest insurance companies in America. </em></p> </section> AI initiatives are driving demand for curated, controlled data. Leaders must move beyond capacity planning and build the governance and quality practices that make data usable. https://cdn.ttgtmedia.com/rms/onlineimages/code_g1195673150.jpg https://www.techtarget.com/data-technologies/tip/Is-your-data-infrastructure-ready-for-the-zettabyte-era Wed, 29 Jul 2026 11:10:00 GMT <p>Who is responsible when an AI agent acts on a supplier proposal that is three years out of date? The developer will likely claim that the agent performed well, but with hopelessly inadequate information. The business will discover that the problem isn't the agent or the model; it's the context, and that redefines the role of the Chief Data Officer.</p> <p>The rise of agentic AI has shifted the chief data officer (CDO) from a traditional data steward to a context curator – the executive accountable for governing what AI systems retrieve and under what definitions.</p> <p>This reorientation of the CDO's role should be exciting, but it faces some genuine challenges. <a href="https://newsroom.ibm.com/2025-11-13-ibm-study-chief-data-officers-redefine-strategies-as-ai-ambitions-outpace-readiness">IBM research</a> shows that 81% of CDOs are focusing on AI investments, but only 26% feel confident that their data can support new revenue streams with AI. A comparable share say their organizations are prepared to use unstructured data to deliver business value.</p> <section class="section main-article-chapter" data-menu-title="The CDO and agentic AI"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The CDO and agentic AI</h2> <p>When AI models were primarily driving chatbots or text generation, <a href="https://www.techtarget.com/searchenterpriseai/tip/Why-does-AI-hallucinate-and-can-we-prevent-it">errors and hallucinations</a> were largely mitigated by human users who interacted with the system. But today, the pressure on CDOs originates in <a href="https://www.techtarget.com/searchenterpriseai/feature/Real-world-agentic-AI-examples-and-use-cases">what agentic systems can do</a>. For teams used to delivering enterprise data to human decision-makers through reports and dashboards, the autonomy of AI agents is a different problem entirely.</p> <p>An AI agent given a business goal -- rebalancing the quarterly marketing budget, drafting a compliance report, routing a vendor contract -- breaks that goal into subtasks, retrieves information from across enterprise systems and executes actions without waiting for a human to evaluate each step. As a result, errors in an agentic workflow compound across steps and may not surface until the action is already complete.</p> <p>The CDO's traditional toolkit was not built for this. Conventional data governance answered specific questions: what data exists, where does it live, is it accurate, and who is authorized to see it? Data catalogs and <a href="https://www.techtarget.com/searchdatamanagement/definition/master-data-management">master data management</a> &nbsp;<a href="https://www.techtarget.com/searchdatamanagement/definition/master-data-management"></a>programs answered those questions well for structured data in reporting environments, where a human analyst received the output and applied judgment.</p> <p>Context architecture for agentic AI must answer a different question: which data, in what combination, with what business definitions, should an AI system retrieve at the time of query to produce a correct result? Context is the defining challenge here.</p> </section> <section class="section main-article-chapter" data-menu-title="Context and the CDO"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Context and the CDO</h2> <p>A data catalog shows users that policy documents exist in a given folder. It does not tell an AI agent which of several versions of that document is authoritative, or that the finance team's definition of revenue excludes deferred recognition while the sales team's does not. Increasingly, CDOs must deliver and maintain a <a href="https://www.techtarget.com/searchenterpriseai/tip/Advantages-of-a-semantic-layer-for-enterprise-AI">semantic layer to capture those distinctions</a>.</p> <p>For example, an agent reviewing a supplier evaluation may read from an enterprise knowledge base that has versions from both 2023 and 2026. How does it know that the 2026 version is authoritative, especially if it has not been given a rule to prefer the most recent version? The agent may use both versions to draft a report that's clear, well-written and professionally formatted. The output contains no hallucinations: the agent executed perfectly, but on <a href="https://www.techtarget.com/searchdatamanagement/news/366646252/Context-is-king-as-agents-evolve-but-problems-persist">unvalidated context</a>. The distinction is important because these two kinds of issues point to different parts of the organization.</p> <p>Accountability for AI outputs depends on someone owning the context layer. In too many enterprises, no single executive owns that authority. If marketing spins up customer sentiment agents while finance builds forecasting agents, then each may apply different definitions of the same business terms, different standards for what counts as an authoritative source and different thresholds for data freshness.</p> <p>The CDO's concern is governing what agents retrieve, a business-critical mandate defined by the following four capabilities:</p> <ul type="disc" class="default-list"> <li><b>Inventory.</b> An enterprise ontology or <a href="https://www.techtarget.com/searchenterpriseai/definition/knowledge-graph-in-ML">knowledge graph</a> maps every data asset while recording relationships between documents, databases, agents, and the sources that feed them. This may include vector databases or unstructured data that traditional data catalogs did not reach. Agents should not retrieve what they cannot understand.</li> <li><b>Access control.</b> Agents can accidentally combine information that humans would never be permitted to view in a single session. Application-layer security, such as the permissions Salesforce enforces on CRM records or SharePoint enforces on documents, governs what a human user can open when they log into a specific system. But an AI agent may simultaneously retrieve and assemble data from multiple systems. Context-layer access control prevents an agent from combining data that no individual human would have been permitted to combine.</li> <li><b>Governed delivery.</b> Context must reach agents without centralizing every system. Model Context Protocol (MCP) -- an open standard with rapidly growing adoption across AI vendors -- enables an agent to query enterprise data sources and receive governed, logged and auditable responses in real time, without the organization first moving all that data into a centralized store.</li> <li><b>Observability.</b> Continuous monitoring of how agents behave, what they retrieve and why they make specific choices. Of the four capabilities, this one most deserves a close look.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Observability"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Observability</h2> <p>For both auditability and continuous development, organizations must understand why an agent made particular choices. Traditional data observability, tracking data quality as it moves through pipelines, and standard LLM observability, tracking prompts and token usage, are no longer sufficient on their own. The table below shows the differences between these forms of observability, &nbsp;illustrating the changing demands on the CDO.</p> <p></p> <p><iframe title="" aria-label="Table" id="datawrapper-chart-Eajgi" src="https://datawrapper.dwcdn.net/Eajgi/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="336" data-external="1"></iframe> <script type="text/javascript">(function(){function e(){window.addEventListener(`message`,function(e){if(e.data[`datawrapper-height`]!==void 0){var t=document.querySelectorAll(`iframe`);for(var n in e.data[`datawrapper-height`])for(var r=0,i;i=t[r];r++)if(i.contentWindow===e.source){var a=e.data[`datawrapper-height`][n]+`px`;i.style.height=a}}})}e()})();</script> </p> <p></p> </section> <section class="section main-article-chapter" data-menu-title="The semantic layer"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The semantic layer</h2> <p>The semantic layer translates governed data into AI-readable context and is critical to all four capabilities. It tells the agent retrieving revenue data whether that term means gross revenue, net revenue or revenue recognized under the organization's specific accounting policy. The semantic layer explicitly records these distinctions, makes them consistently available across data systems and delivers data to the agent with that precision intact. Without semantics, <a href="https://www.techtarget.com/searchdatamanagement/feature/Why-enterprise-AI-depends-on-the-semantic-layer">agents guess the meaning of business terms</a> based on their model's general training, rather than on specific business vocabulary.</p> <p>As Fern Halper, senior research director of TDWI, put it, effective organizations "formalize an enterprise business vocabulary and explicitly encode business rules and logic so that models operate within defined guardrails rather than inferred assumptions." Human validation of those definitions remains part of that process, because the semantic layer is only as reliable as the business logic people have approved and recorded in it. If an agent measures its performance against KPIs that influence operational or strategic decisions, someone needs to verify that those KPIs are properly defined and managed.</p> </section> <section class="section main-article-chapter" data-menu-title="The new CDO"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The new CDO</h2> <p>CDOs who can build and manage this infrastructure of context transform their businesses. An enterprise with a mature context layer dramatically shortens the time required to deploy new agents safely. The CDO who can own this infrastructure becomes the executive who manages what AI knows about the business. That sets the boundary of what the enterprise is capable of.</p> <p><em>Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> Agentic AI demands that CDOs move beyond cataloging to curating the context AI systems retrieve, creating a new accountability layer for data governance. https://cdn.ttgtmedia.com/rms/onlineimages/folder-files13.jpg https://www.techtarget.com/data-technologies/feature/The-CDOs-new-role-is-curating-context-for-data-governance Wed, 29 Jul 2026 11:00:00 GMT <p>Nimble on Wednesday launched Web Search Agents, new agentic AI applications that autonomously learn customer use cases to scour the internet for relevant information that can be converted into curated data tables.</p> <p>While analytics and AI tools are largely informed by an enterprise's operational data, information from the web is also valuable when building applications that inform decisions and business processes. Operational data reveals what is happening within an organization, while web data shows what is taking place externally.</p> <p>Together, the two provide <a href="https://www.techtarget.com/searchdatamanagement/opinion/Context-is-the-make-or-break-layer-for-AI-in-production">richer context for agents</a> and other decision-informing tools than one type of data alone.</p> <p>"Web data is the world's largest real-time signal network," Michael Ni, an analyst at Constellation Research, told TechTarget. "Nimble allows enterprises to combine internal and external data, enabling AI to move from internally informed analysis to context-aware decisions."</p> <p>Use cases include real time competitive intelligence, such as dynamic pricing and product availability, examining market and sentiment signals, insight into competitors, and <a href="https://www.techtarget.com/searchenterpriseai/tip/Global-AI-legislation-and-regulation-tracker">ensuring regulatory compliance</a>.</p> <p>Nimble's Web Search Agents are designed to automate much of the research work that previously had to be performed by humans, building and maintaining complex <a href="https://www.techtarget.com/whatis/feature/How-to-scrape-data-from-a-website">web-scraping tools</a>. In addition, Nimble's agents are designed to self-learn a user's domain to perform web data research with the accuracy of an expert, but with greater efficiency.&nbsp;&nbsp;</p> <p>As a result, Web Search Agents are a potentially significant addition for Nimble customers, according to Donald Farmer, founder and principal of TreeHive Strategy.</p> <p>"There has always been a bottleneck accessing web data because it meant building and maintaining brittle scraping scripts and managing proxies," he told TechTarget. "Agents should autonomously … remove that bottleneck."</p> <p>Based in New York City, Nimble is a web data search specialist that <a href="https://www.techtarget.com/searchdatamanagement/news/366639416/Web-data-curator-Nimble-secures-47M-to-fuel-growth">raised $47 million</a> in venture capital funding in February to fuel its growth. In addition to Nimble, vendors that specialize in enabling users to pull data from the web include Apify, Bright Data, Grepsr, MixRank and&nbsp;<a href="https://www.computerweekly.com/blog/Data-Matters/Giving-ourselves-the-chance-to-lead-the-AI-race-and-stay-ethical-too">Oxylabs</a>.</p> <section class="section main-article-chapter" data-menu-title="Aiding AI"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Aiding AI</h2> <p>Agents and other AI tools require reams of contextually relevant information to deliver accurate outputs at a high enough rate that they can be <a href="https://www.techtarget.com/searchcio/feature/Startup-founder-says-trust-is-biggest-barrier-to-AI-agents">trusted in production</a>.</p> <p>Despite continually <a href="https://kpmg.com/us/en/media/news/q1-ai-pulse2026.html">rising investments</a> in AI development, <a href="https://www.deloitte.com/us/en/what-we-do/capabilities/applied-artificial-intelligence/content/state-of-ai-in-the-enterprise.html?id=us:2ps:3gl:aisgm26:awa:CONS:em:K0218784:012626:kwd-430833501819:195648817121:794247818306::&amp;gclsrc=aw.ds&amp;gad_source=1&amp;gad_campaignid=23269751971&amp;gclid=CjwKCAjw6MPRBhBTEiwAd-7Mrwla_c_gzzSu2TNPVBWdBS4INV4yNdYgtukO2OU2-zWBAuwZE-s3mBoCwegQAvD_BwE">more projects fail</a> than succeed. Frequently, difficulties discovering and connecting agents with the context that enables them to perform properly &nbsp;prevent AI initiatives from moving beyond the pilot stage.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> There has always been a bottleneck accessing web data because it meant building and maintaining brittle scraping scripts and managing proxies. Agents should autonomously … remove that bottleneck. </figure> <figcaption> <strong>Donald Farmer</strong>Founder and principal, TreeHive Strategy </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>With context critical to developing trustworthy AI tools, many vendors have introduced products in 2026 designed to help customers discover and operationalize relevant data. In June alone, <a href="https://www.techtarget.com/searchdatamanagement/news/366644853/AWS-latest-to-introduce-context-layer-for-agentic-AI">AWS</a>, <a href="https://www.techtarget.com/searchdatamanagement/news/366644813/Databricks-intros-new-Genie-data-management-tools-to-aid-AI">Databricks</a>, <a href="https://www.techtarget.com/searchdatamanagement/news/366643955/Microsoft-boosts-Fabric-to-make-it-a-foundation-for-AI">Microsoft</a> and <a href="https://www.techtarget.com/searchdatamanagement/news/366643795/Snowflake-barrage-adds-more-AI-development-analysis-tools">Snowflake</a> unveiled new features aimed at connecting agents with context.</p> <p>With Web Data Agents, Nimble is similarly providing capabilities aimed at helping customers feed agents with contextually relevant data, in a move motivated by user feedback, according to Uri Knorovich, the vendor's co-founder and CEO.</p> <p>"The feedback we kept hearing was simple -- [customers'] agents could search the web, but they weren't getting the specific data they needed. … Agents need web search that adapts to their specific task, context and objectives to optimize for accuracy and cost, not one-size-fits-all results," he said.</p> <p>Web Search Agents are built on customized search algorithms that enable them to learn autonomously, Knorovich continued.</p> <p>"These advances allow search models to continuously learn a domain, optimize retrieval for a specific use case and deliver … more accurate web context than generic search engines," he said.</p> <p>Beyond accurately finding and retrieving relevant web data, Nimble's Web Search Agents are designed to <a href="https://www.techtarget.com/searchenterpriseai/tip/Reduce-project-expenses-with-AI-cost-optimization-strategies">reduce the cost</a> of searching the internet for task-specific context.</p> <p>Using generic search engines that are task-agnostic can become expensive, according to Farmer, who noted that such tools often return massive, unstructured amounts of text that, when processed by an agent, consumes a high number of <a href="https://www.techtarget.com/whatis/definition/context-window">context window</a> tokens.</p> <p>In benchmark testing conducted by Nimble but not independently verified, Web Search Agents increased answer quality by over 20% and approximately halved the number of tokens spent on each query.</p> <p>"A smart agent retrieving data should therefore be a great advantage," Farmer said.</p> <p>However, while valuable for Nimble customers, Web Search Agents are not the only such web data discovery and retrieval tools on the market, he continued.</p> <p>For example, Grepsr and MixRank both offer AI-powered web scraping capabilities that can discover and deliver task-specific data. However, Farmer pointed out that there is slight differentiation, since Nimble's Web Search Agents deploy <a href="https://www.techtarget.com/whatis/definition/headless-browser">headless browsers</a> that pull the freshest version of a web page instead of a data <a href="https://www.techtarget.com/searchstorage/definition/cache">cache</a>.</p> <p>"If you need that, it is a compelling feature," Farmer said. "I don't think there's a lot here that is innovative compared to the rest of the market, but the work appears to be solid."</p> <p>Ni similarly noted that Nimble's web data collection capabilities are not unique. Instead, the vendor has the potential to distinguish itself by combining live access to web data, agent-powered navigation and repeatable output into a managed external sensing layer, he continued.</p> <p>"The proof point will be whether that delivers greater completeness and lower maintenance costs than traditional scraping platforms, vertical data platforms, or AI-native search APIs," Ni said.</p> </section> <section class="section main-article-chapter" data-menu-title="Next steps"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Next steps</h2> <p>While operational data remains the focal point for many enterprises, Nimble's leadership believes that web data is becoming the primary source of <a href="https://www.techtarget.com/searchenterpriseai/tip/Exploring-the-context-layer-for-AI-systems">context for agents</a>, according to Knorovich.</p> <p>With that in mind, Nimble plans to make it easier to connect an enterprise's agents with web search by adding and improving self-learning and agent memory capabilities so Web Search Agents better understandtasks and get smarter with each use.</p> <p>"The bigger vision is a web that agents can navigate as fluently as people do, and we intend to keep building toward that," Knorovich said.</p> <p>Ni noted that Nimble is positioning itself as an enterprise-grade layer for providing external context to agents. Going forward, its opportunity to continue serving existing users and attract new ones is by turning that layer into <a target="_blank" href="https://www.linkedin.com/pulse/rise-ai-control-planes-governing-models-agents-scale-mahmoud-abufadda-d1f5f/" rel="noopener">a control plane</a> for external context.</p> <p>"Parallel to the growth of internal context layers, enterprise leaders will want to know where any specific fact came from, how fresh and reliable it is, whether sources conflict, and whether an agent is permitted to act on it given a specific context," Ni said. "Productizing context controls would expand Nimble’s relevance from developers to CDAOs and CAIOs."</p> <p><i>Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than three decades of experience. He covers analytics and data management.</i></p> </section> The vendor's Web Search Agents autonomously learn domain-specific use cases to help enterprises collect contextually relevant information for AI and analytics. https://cdn.ttgtmedia.com/rms/onlineimages/telecommunications_g1189468316.jpg https://www.techtarget.com/data-technologies/news/366646240/Nimble-launches-agents-to-cull-relevant-web-data-for-AI Wed, 29 Jul 2026 09:00:00 GMT 60 webmaster@techtarget.com