Copyright TechTarget - All rights reserved https://cyber.law.harvard.edu/rss/rss.html Techtarget Feed Generator en Fri, 17 Jul 2026 13:44:20 GMT https://www.techtarget.com/searchdatamanagement editor@techtarget.com <p>As enterprises move AI initiatives from pilots to production deployments of autonomous agents, attention is shifting to the data architectures needed to support those systems.</p> <p>Data fabrics have emerged as one approach for connecting data across repositories while applying consistent governance, business context and access controls. The latest iteration is being shaped by the <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">need to help AI agents work across enterprise systems</a> without forcing organizations to replace existing data lakes, lakehouses, pipelines or mesh deployments.</p> <p>That can benefit organizations operating on tight budgets and looking to maximize existing data infrastructure. There's another potential bonus: A fabric's contextual layer could become increasingly important as the enterprise challenge shifts from connecting data to helping AI understand the business behind the data.</p> <p>"That's why many organizations view AI data fabrics as the next evolution of traditional data architectures, rather than a replacement for them," said Rima Safari, U.S. Data, Analytics and AI Practice Leader at PwC.</p> <p>As agent adoption grows, data leaders must decide whether a data fabric can reduce integration complexity, improve observability and help AI systems operate at scale. They should also pay special attention to flexibility, since a data fabric opens opportunities to accommodate changes to data sources or retrieval methods.</p> <section class="section main-article-chapter" data-menu-title="Why AI data fabrics are drawing attention"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why AI data fabrics are drawing attention</h2> <p>Agentic AI has made architectural decisions more urgent. CIOs and other technology leaders face a <a href="https://www.techtarget.com/searchcio/feature/the-ai-agent-governance-gap-how-cios-can-gain-control">complex management challenge</a> as more autonomous agents move into production.</p> <p>The modern AI data fabric extends beyond data integration by adding semantic context and governance layers needed for these autonomous agents. Potential benefits include support for data products, flexibility to adapt to changing technologies and improved observability.</p> <p>"You need some kind of an enterprise data fabric to be the path between the agents that people are trying to build and all of that enterprise data that is potentially siloed," said Kevin Keuning, a senior vice president at NWN, a technology services provider.</p> <p>This AI-oriented fabric model expands the role of earlier data fabric designs.</p> <p>"Traditional data architectures focus on storing, moving and organizing data," Safari said. "An AI data fabric adds the context, intelligence and governance that AI systems and agents need to operate effectively across the enterprise."</p> </section> <section class="section main-article-chapter" data-menu-title="How data fabrics help agents do their jobs"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How data fabrics help agents do their jobs</h2> <p>AI data fabrics can help facilitate agentic AI rollouts through a centralized governance layer that spans agents and the data they use.</p> <p>"Without that kind of a layer, you would need to look at each individual silo of data and then determine the right path for agents taking advantage of that data," Keuning said.</p> <p><a href="https://www.techtarget.com/searchenterpriseai/tip/Exploring-the-context-layer-for-AI-systems">AI data fabrics also serve up business context</a>, which is critical for agents making decisions or recommendations. To do that, the architecture can apply a common business vocabulary on top of an enterprise's domains -- a bank's mortgage or credit card businesses, for instance -- so an AI agent can understand the data within those domains, said Pramod Sadalage, Distinguished Engineer at Thoughtworks, a technology consulting company.</p> <p>This vocabulary includes domain concepts, definitions, relationships, policies and business meaning, supported by a <a href="https://www.techtarget.com/searchdatamanagement/feature/Why-enterprise-AI-depends-on-the-semantic-layer">semantic layer</a>, he said. This approach gives people and AI agents a more consistent way to interpret enterprise data, he added.</p> <p>But Sadalage pointed to a crucial difference between the two.</p> <p>"When you put data in front of humans, they have a bunch of context in their heads, being in the business domain for some time, and they understand what it means," he said. "But the AI agent will struggle to understand what the data means if you don't have the right context around it."</p> <p>An AI data fabric also plays a role in managing identities and access rights as the agent population grows.</p> <p>"In most cases, the AI data fabric layer can be pretty helpful in making that possible," Keuning said. "That's the layer where we can define that governance around the identity of the human end user, or the agentic end user or a mix of both. That has to be part of the whole architecture."</p> <p>Sadalage also cited the issue of access rights. An AI agent granted authority beyond a user's privileges could expose data the user is not authorized to see. For that reason, the user's authentication and privileges should be delegated to the agent, he noted.</p> <p>Delegation happens at the agent layer, where the agent acts on behalf of the user. The fabric responds to the agent as though the user were requesting the data, Sadalage said.</p> <p>Data fabrics generally play well with other architectures.</p> <p>Keuning said AI data fabrics and data meshes, for instance, are complementary. Data mesh distributes data ownership and governance by domain. Data fabrics, meanwhile, can provide a unified governance and access layer across those resources to support AI deployments, he noted.</p> <p>"The data mesh handles local governance within each domain, and the fabric handles global governance across all domains," he said.</p> <p>Some organizations are already combining data fabric and data mesh principles within a single architecture. For example, Stibo Systems built an internal enterprise intelligence platform <a href="https://www.techtarget.com/searchdatamanagement/news/366643955/Microsoft-boosts-Fabric-to-make-it-a-foundation-for-AI">on Microsoft Fabric</a> that combines centralized governance and shared semantic models with data mesh principles, such as domain-level ownership and reusable data products.</p> <p>Thomas Møgelmose, CFO at Stibo Systems, described the model as "a centralized, governed platform combined with decentralized ownership and data product thinking."</p> </section> <section class="section main-article-chapter" data-menu-title="Building flexibility for evolving AI architectures"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Building flexibility for evolving AI architectures</h2> <p>On the storage side, primary repositories within a data fabric can include <a href="https://www.techtarget.com/searchdatamanagement/tip/Assemble-the-layers-of-big-data-stack-architecture">data lakes and data lakehouses</a>, Keuning said. Other data might enter an organization through APIs, he added.</p> <p>An AI data fabric serves as an abstraction layer that provides adaptability.</p> <p>"The fabric exposes a stable governance interface over the data while the underlying systems can change," Keuning said.</p> <p>In the case of agentic AI, an organization might need to optimize data for vector search or <a href="https://www.techtarget.com/searchdatamanagement/tip/5-knowledge-graph-use-cases-in-data-fabric-architecture">knowledge graphs</a>, Keuning said. Those techniques help AI agents retrieve relevant information with semantics and support reasoning.</p> <p>That flexibility matters as organizations test different ways to connect agents with enterprise data. Keuning cited agentic retrieval-augmented generation, keyword search and other types of agentic search as examples.</p> <p>That evolution will continue, as will the need to adjust.</p> <p>"There will be more approaches," Keuning said. "Having that optionality in the architecture, so that you can change the way data stores are either optimized or presented to the agents, is going to be important."</p> </section> <section class="section main-article-chapter" data-menu-title="Obstacles to AI data fabric adoption"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Obstacles to AI data fabric adoption</h2> <p>Data leaders must consider implementation challenges that could hinder a fabric strategy and its expected benefits.</p> <p>Sadalage pointed to integration complexity as a potential deployment issue for AI data fabrics.</p> <p>"The biggest challenge I see is there is no one tool that says, 'Hey, this is the AI data fabric,'" he said.</p> <p>As a result, organizations might need to pull together an enterprise <a href="https://www.techtarget.com/searchdatamanagement/feature/16-top-data-catalog-software-tools-to-consider-using">data catalog</a>, data integration tools, <a href="https://www.techtarget.com/searchsecurity/tip/Secure-MCP-servers-to-safeguard-AI-and-corporate-data">Model Context Protocol servers</a>, a semantic layer and metadata management platforms. Businesses accustomed to creating data products will likely have some of those components. Even so, they could still face obstacles, such as making their data AI-ready, Sadalage said.</p> <p>Safari identified the semantic layer as the "real differentiator" of an AI data fabric. That layer consists of knowledge graphs, ontologies and business context that help AI systems understand how data relates across the organization, she said. But creating a shared business context through metadata and semantic models can prove a big hurdle, she added.</p> <p>Overall, the technology challenge is manageable, Safari noted. Structural considerations can loom large, however.</p> <p>"The bigger challenge is creating trust, governance, and organizational alignment at enterprise scale," she said. "Most organizations don't struggle because they lack data. They struggle because data is fragmented across systems, functions, and ownership models."</p> <p>In that context, success depends on integrating legacy and modern platforms, she added.</p> </section> <section class="section main-article-chapter" data-menu-title="A growing need for observability"> <h2 class="section-title"><i class="icon" data-icon="1"></i>A growing need for observability</h2> <p>Data managers should plan to increase their focus on monitoring and tracing AI-driven workflows.</p> <p>"With AI agents in the mix now, people are asking for a lot more observability," Sadalage said. "If there is some decision being made by the agent, I need to know what question was asked, what data was it looking at, when was it looking at it and what decision did it make."</p> <p>Those questions become important when auditors ask about the <a href="https://www.techtarget.com/searcherp/feature/AI-decision-trails-are-the-new-audit-trail">AI decision trail</a>.</p> <p>Keuning cited observability as a fabric deployment challenge that can become a benefit if properly implemented. He said enterprises face a transition similar to that of cloud computing.</p> <p>"When we built a cloud-native architecture, all of our observability tools were impacted," Keuning said. "To some extent, that is happening again with this AI-native architecture."</p> <p>He said some tools are breaking or weren't designed to work with newer fabric layers. Agent <a target="_blank" href="https://mitsloan.mit.edu/ideas-made-to-matter/agentic-ai-explained" rel="noopener">loops</a>, for example, introduce components that existing observability tools were not built to monitor, he noted.</p> <p>Overcoming the observability obstacle could give organizations an edge.</p> <p>"If we have an issue with slow response times, data leakage or governance violations, we need to be able to trace that all the way through the layers of this architecture to its root cause," Keuning said.</p> <p><em>John Moore is a freelance writer who has covered business and technology topics for 40 years. He focuses on enterprise IT strategy, AI adoption, data management and partner ecosystems.</em></p> </section> The latest take on data fabric architecture promises to help AI agents coexist with existing platforms, but there's some assembly required. https://cdn.ttgtmedia.com/rms/onlineimages/strategy_g1151390404.jpg https://www.techtarget.com/searchdatamanagement/tip/AI-data-fabric-emerges-as-a-governance-layer-for-agents Fri, 17 Jul 2026 08:11:00 GMT <p>Agentic AI isn't just reshaping analytics and operational processes in organizations; it's also sharpening familiar data management challenges and introducing new ones.</p> <p>AI agents amplify traditional data quality and governance problems -- including issues with data completeness, consistency, lineage and access. But they also pose specific challenges involving context, semantics, bias and data pipelines.</p> <p>In addition, agentic AI significantly expands the potential blast radius of data issues. When bad numbers surface in a dashboard, a data analyst can catch them before the daily morning meeting. But when an AI agent autonomously analyzes the same data and acts on its conclusions across a dozen systems within seconds, the window for catching the errors slams shut and they rapidly propagate across the organization.</p> <p>Without a solid data foundation, AI agents likely will remain confined to pockets of productivity rather than delivering meaningful business value. To make agentic AI <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-prepare-your-business-for-agentic-AI-adoption">effective at enterprise scale</a>, data leaders must invest in processes and guardrails to overcome the following key challenges.</p> <section class="section main-article-chapter" data-menu-title="1. Maintaining high data quality levels"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Maintaining high data quality levels</h2> <p>Poor data quality is the leading cause of AI project failures, said Doug Gilbert, CIO and chief digital officer at digital transformation consultancy Sutherland. Agentic AI applications exacerbate <a href="https://www.techtarget.com/searchenterpriseai/feature/9-data-quality-issues-that-can-sideline-AI-projects">data quality issues</a>, especially if agents are configured to act autonomously.</p> <p>"When the underlying data is incomplete, inconsistent, outdated, biased or ungoverned, the risks go far beyond mediocre outputs," Gilbert said. "They trigger cascading errors at machine speed in critical business applications."</p> <p>Data management best practices are shifting accordingly, he added. Master data management is moving from an optional discipline to a foundational one for ensuring data quality, although MDM initiatives themselves remain challenging due to their technical complexity and high implementation costs. Organizations are also increasingly embedding <a href="https://www.techtarget.com/searchdatamanagement/opinion/Data-observability-for-AI-helps-curb-poor-model-performance">data observability and quality gates</a> directly into data pipelines rather than auditing them after the fact, according to Gilbert.</p> <p>Data quality fundamentals must be at the front of the conversation when organizations <a href="https://www.techtarget.com/searchenterpriseai/feature/Real-world-agentic-AI-examples-and-use-cases">deploy AI agents</a>, noted Rosaria Silipo, a data scientist, author and co-host of the "My Data Guest" podcast. "An agent is only as reliable as the data it reasons on, and that is still a human responsibility," she said.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Stopping 'garbage agents' from scaling"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Stopping 'garbage agents' from scaling</h2> <p>Dan Federoff, vice president and head of data at IT consultancy Bridgenext, said agents won't question or push back on bad data. The danger, he warned, is they'll spread it across the organization through flawed analytics and actions before anyone notices.</p> <p>This has inspired a variant of the traditional "garbage in, garbage out" adage in data management circles: "garbage data, garbage agents." In addition to <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-for-AI-requires-a-cross-functional-approach">strong data quality, lineage and governance processes</a>, standardized prompting and clear engineering guardrails are necessary to prevent errors from proliferating, Federoff said. Managed effectively, agents safely scale execution within those boundaries, he added.</p> </section> <section class="section main-article-chapter" data-menu-title="3. Providing sufficient context about data to agents"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Providing sufficient context about data to agents</h2> <p>To function accurately, AI agents also <a href="https://www.techtarget.com/searchenterpriseai/tip/Exploring-the-context-layer-for-AI-systems">require a context layer</a> built on metadata and shared semantics that provides a clear understanding of data relationships and rules, according to Gartner. Without one, agents "are far more likely to hallucinate, introduce bias and produce unreliable results," Gartner analyst Rita Sallam said during a session at the Gartner Data &amp; Analytics Summit in London in May 2026.</p> <p>By 2027, Gartner predicted, organizations that prioritize developing unified semantics will increase agent accuracy by up to 80% and reduce agentic AI costs by up to 60%. Reducing errors and increasing trust in data for autonomous decision-making should <a href="https://www.techtarget.com/searchdatamanagement/feature/Why-enterprise-AI-depends-on-the-semantic-layer">push data leaders to invest in semantics</a> "as a non-negotiable foundation" for agentic AI, Sallam said. She also expects regulators to demand greater transparency about the semantics that agents use.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Controlling data access by agents"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Controlling data access by agents</h2> <p>Steve Touw, co-founder and CTO at data security platform vendor Immuta, said traditional identity and access management approaches <a href="https://www.techtarget.com/searchsecurity/tip/Cybersecuritys-agentic-AI-identity-crisis-and-how-to-fix-it">fail to scale sufficiently for agentic AI</a>. Trying to solve data security issues at the authentication level by simply proving an agent's identity is a mistake, Touw said. The real problem is authorization: governing in real time what data the agent is allowed to access.</p> <p>When an organization has hundreds or thousands of agents acting on behalf of various users with different permission levels, managing agent permissions manually or through static roles is impossible, according to Touw. Ghost accounts and over-privileged agents become a ticking time bomb for data leaks.</p> <p>"You have to move from gatekeeping access to orchestrating it," Touw said. He recommended decoupling security policies from the data storage layer. An abstract authorization tier ensures that each user's permissions travel with an agent, enabling data teams to automatically enforce fine-grained controls -- such as data masking or row-level security -- regardless of which system the agent pulls data from. &nbsp;</p> </section> <section class="section main-article-chapter" data-menu-title="5. Preventing agents from amplifying data bias"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Preventing agents from amplifying data bias</h2> <p>Data bias is a <a href="https://www.techtarget.com/searchenterpriseai/feature/6-ways-to-reduce-different-types-of-bias-in-machine-learning">common issue in AI</a>, and agentic systems compound the risks for organizations. If mismanaged, "every cache between your data and your agent is a lie waiting to be narrated," said Rock Lambros, director of AI security and governance at software vendor Zenity. Bias in training and production datasets similarly creates the potential for erroneous or discriminatory outcomes to go unnoticed, which can negatively affect business decisions and damage a company's reputation.</p> <p>In addition to <a href="https://www.techtarget.com/searchenterpriseai/feature/The-AI-bias-playbook-Mitigation-strategies-for-CIOs">rooting out data bias</a>, Lambros recommended treating an agent's session history as a data management problem. Old results can bias future analysis or decisions, he noted. For example, Lambros has seen an agent skip a valid data source because a process failed an hour earlier, then create a confident answer to an analytics query from incomplete data -- or worse, nothing. The fix, he said, is periodic session resets combined with tracking provenance metadata for every artifact the agent reads.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Managing large amounts of unstructured data"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Managing large amounts of unstructured data</h2> <p>Agentic AI tools unlock greater access to unstructured data, which they can use for new types of analytics. Indeed, Stephen Catanzano, a senior analyst at Informa TechTarget's Omdia division, wrote in a March 2026 TechTarget <a href="https://www.techtarget.com/searchdatamanagement/opinion/Why-agentic-AI-demands-both-structured-and-unstructured-data">article</a> that AI agents often need to access unstructured data to get contextual information that conventional structured data doesn't provide. Otherwise, he said, their reasoning and decisions will be based on an incomplete understanding of the available data.</p> <p>However, many organizations have vast repositories of unstructured data that are <a href="https://www.techtarget.com/searchdatamanagement/post/Unstructured-data-is-the-bottleneck-for-agentic-AI-success">harder to wrangle and manage</a> than structured datasets. Catanzano recommended deploying a data lakehouse that supports unified governance and management of both structured and unstructured data. Krishna Subramanian, president and COO at unstructured data management vendor Komprise, said data teams should also classify unstructured data at scale to prevent misuse by agents while enabling them to discover and query the right data for specific applications.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Supporting bidirectional data integration"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Supporting bidirectional data integration</h2> <p>AI agents are fundamentally changing data integration. Conventional integration for analytics applications is a one-way pipeline: data moves from source systems into a data warehouse or other repository.&nbsp;"AI agents break that model because they don't just consume integrated data," said David DuChene, senior manager of data and AI professional services at SHI International. "They generate new data, discover latent relationships and write enriched context back into the data estate."</p> <p>This means data integration must become bidirectional and, in many applications, continuous. To enable that, DuChene said data teams should embed data governance into the integration architecture and develop an entity resolution layer that creates a unified view of related records across different data sources. He also recommended taking a domain-by-domain approach rather than an enterprise one. Doing so will help organizations find AI-driven business value "in weeks rather than years," he said.</p> <p><em>George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.</em></p> </section> AI agents pose new data management challenges and magnify familiar ones. To build a solid agentic foundation, data leaders must plan for these critical challenges. https://cdn.ttgtmedia.com/visuals/searchEnterpriseAI/ai-enterprise-apps/searchEnterpriseAI_029.jpg https://www.techtarget.com/searchdatamanagement/feature/How-agentic-AI-amplifies-data-management-challenges Thu, 16 Jul 2026 15:54:00 GMT <p>AI adoption has revealed a structural gap in enterprise architecture: AI systems excel at pattern recognition but struggle with consistent reasoning across business domains.</p> <p>Without a formal definition of data's meaning, AI infers context rather than using governed business logic. This can lead to <a href="https://www.techtarget.com/searchenterpriseai/feature/5-AI-risks-businesses-must-confront-and-how-to-address-them">hallucinations and unreliable outputs</a>, resulting in poor business decisions that increase the company's financial, reputational and legal risk.</p> <p>Trustworthy AI outputs require a common business language. A data ontology is a formal, machine-readable framework that defines business concepts, their relationships and the rules systems can use to draw conclusions from those relationships. Many companies don't have an ontology -- largely because their data is disorganized.</p> <section class="section main-article-chapter" data-menu-title="Data silos undermine AI reliability"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Data silos undermine AI reliability</h2> <p>A July 2024 Gartner <u><a target="_blank" href="https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk" rel="noopener">survey</a></u> found that 63% of organizations either lack or are unsure they have the right data management practices for AI. Unreadiness has severe consequences. Gartner forecasts 60% of AI projects will be scrapped due to a lack AI-ready data by the end of 2026. This prediction applies directly to organizations deploying AI agents without ontology infrastructure.</p> <p>Most enterprises <a href="https://www.techtarget.com/searchstorage/opinion/IT-leaders-face-data-infrastructure-gaps-as-AI-workloads-grow">store data across several independent systems</a> that were never designed to communicate with each other. When an AI agent queries across those sources, it may lack the governed mechanism to reconcile conflicting records or detect duplicates.</p> <p>A data ontology provides a shared semantic target that each system can map to. It formally defines a "customer" through attributes, identifiers and relationships with other business entities. For example, in a non-ontological platform, AI may not recognize a "customer" in <a href="https://www.techtarget.com/searchcustomerexperience/tip/What-are-the-benefits-and-challenges-of-CRM">a CRM platform</a>, a "counterparty" in fraud detection and a "subject" in a compliance system can refer to the same real-world entity.</p> <p>With an ontology, the CRM system, fraud detection platform and the compliance system each map records to a governed definition. Now, when an AI agent queries "customer," it can work through the ontology layer, which has already reconciled the disconnected records. Even though the physical data systems are separate, the AI has a consistent understanding across sources.</p> </section> <section class="section main-article-chapter" data-menu-title="Inconsistent data definitions weaken AI outputs"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Inconsistent data definitions weaken AI outputs</h2> <p>More organizations are realizing the importance of investing in data semantics. A Futurum Group 2026 <a target="_blank" href="https://futurumgroup.com/press-release/enterprise-data-analytics-survey-finds-59-investing-in-semantic-layers-as-critical-ai-infrastructure/" rel="noopener">survey</a> found that nearly 59% of respondents planned additional spending on semantic layers, with 44.5% planning to increase existing spend and 14.4% plan to adopt for the first time. But distributed data is not the only challenge for reliable AI outputs. Connected systems can also cause confusion.</p> <p>Just as AI has no way of reconciling different terms that fit the same definition, it also has no mechanism for knowing which definition applies to which context. For example, "revenue" may be calculated differently in a CRM system and an ERP system. "Active user" may differ between a product analysis platform and a customer success tool. When data lacks a governed definition, systems can generate outputs that are accurate at the database level but wrong at the business level with no error signal to catch it.</p> <p>Data ontology provides a single, canonical definition for each governed business concept, so terms are no longer defined by each system's own logic. The ontology gives systems and AI agents a shared reference point.</p> <p>The distinction between a semantic layer and an ontology matters. A semantic layer standardizes business metrics and query logic for analytics. An ontology defines what entities are, how they relate and inference rules from those relationships. AI agents need both, but the ontology provides a <a href="https://www.techtarget.com/searchdatamanagement/opinion/Context-is-the-make-or-break-layer-for-AI-in-production">structured context</a> for reasoning.</p> </section> <section class="section main-article-chapter" data-menu-title="How data ontologies support AI reasoning"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How data ontologies support AI reasoning</h2> <p>Connecting distributed data sources and standardizing business meaning are useful foundations, but enterprises need more to optimize AI output.</p> <p>Large language models generate responses based on patterns developed during training, which <a href="https://www.techtarget.com/searchenterpriseai/tip/Generative-AI-vs-predictive-AI-Understanding-the-differences">works well for text generation and summarization</a>. That approach isn't reliable for multi-step business reasoning, when correct answers depend on precise meanings. An AI agent determining whether a transaction clears a compliance threshold or whether a customer qualifies for a pricing tier must apply specific business logic. Statistical patterns don't produce such answers.</p> <p>An ontology adds a <a href="https://www.techtarget.com/searchcio/tip/AI-agents-are-running-wild-Secure-the-reasoning-layer-now">reasoning layer</a>. Where a semantic layer standardizes measures, an ontology encodes relationships and rules that support logical inference.</p> <p>The World Wide Web Consortium's Web Ontology Language (OWL) specification is the formal standard underlying enterprise ontologies. OWL enables programs to verify logical consistency and make implicit knowledge explicit. AI agents grounded in an OWL-based ontology don't estimate whether a transaction meets a threshold; they apply the defined rules and return a verifiable answer.</p> </section> <section class="section main-article-chapter" data-menu-title="Putting data ontologies into practice"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Putting data ontologies into practice</h2> <p>For enterprise teams beginning this work, consider the following steps:</p> <ol class="default-list"> <li><b>Define the business concepts AI agents need to reason about before mapping the data. </b>Start by formally specifying core entities, such as "customer," "product," "revenue" and "transaction." Data mapping follows, giving disconnected systems a shared semantic target.</li> <li><b>Build the ontology before deploying AI agents.</b> Many organizations address semantic grounding only after AI deployments produce unreliable outputs. The ontology must precede the agent. AI agents connected to raw schemas inherit whatever inconsistencies those schemas carry.</li> <li><b>Ground agents in the ontology layer.</b> Relying on prompt engineering forces AI agents to guess at data definitions at inference time, producing inconsistent results. An ontology governs meaning at the data layer and creates consistent, auditable AI outputs across workflows.</li> <li><b>Establish a semantic accuracy baseline before and after ontology grounding.</b> Query execution is a functional test. Semantic accuracy measures whether AI outputs conform to governed business definitions.</li> </ol> <p>The path to reliable AI output runs through a governed data layer. A 2024 <a target="_blank" href="https://arxiv.org/pdf/2405.11706" rel="noopener">study</a> showed that LLMs querying raw SQL databases managed a 16.7% accuracy rate. However, by grounding the model in an ontology-backed knowledge graph, accuracy rose to 54.2%. With an ontology-based query check, accuracy reached 72.55%. Those figures quantify the value of governed semantics in making AI outputs less unpredictable and more useful as enterprise assets.</p> <p><em>Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.</em></p> </section> Data ontologies strengthen AI outputs with governed definitions across platforms and departments, helping data leaders boost trust and improve decision-making. https://cdn.ttgtmedia.com/rms/onlineimages/ai_a311883856.jpg https://www.techtarget.com/searchdatamanagement/tip/Data-ontologies-are-foundational-for-usable-AI-outputs Wed, 15 Jul 2026 15:21:00 GMT <p>The rapid adoption of generative AI tools and agentic AI systems is reshaping how organizations manage and use data.</p> <p>While these technologies can create significant business value, they also expose weaknesses in traditional data governance models built on static policies, manual enforcement and periodic reviews. In fast-moving, distributed data environments that fuel AI applications, governance must become continuous and adaptive.</p> <p>AI <a href="https://www.techtarget.com/searchdatamanagement/opinion/How-AI-powered-governance-enables-scalable-AI-deployment">plays a key role</a> in that, too. When used as part of a data governance program, AI can automatically enforce policies, improve the <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-challenges-that-can-sink-data-operations">scalability of governance processes</a> and address risks in real time. This enables organizations to move from reactive governance to proactive policy enforcement.</p> <p>But AI can't be left to its own governance devices. Successful data governance <a href="https://www.techtarget.com/searchdatamanagement/opinion/Human-oversight-enables-automated-data-governance">still depends on strong human oversight</a> from data governance managers, data stewards and data owners, who must ensure AI-driven enforcement decisions align with data governance policies and internal data standards.</p> <section class="section main-article-chapter" data-menu-title="Why AI is essential for modern data governance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why AI is essential for modern data governance</h2> <p>Traditional data governance approaches worked well when data volumes were smaller and usage patterns were predictable. But many organizations now need to <a href="https://www.techtarget.com/searchdatamanagement/feature/Controlling-data-sprawl-requires-governance-discipline">govern ever-increasing volumes</a> of structured and unstructured data across decentralized operational systems and data platforms. They're also deploying AI systems that <a href="https://www.techtarget.com/searchdatamanagement/opinion/Most-data-governance-wasnt-built-for-AI">continuously generate and consume data</a>, often without direct human control. Governance risks, such as data leakage, privacy breaches and regulatory violations, can emerge quickly in these dynamic environments.</p> <p>Governance policies that are enforced manually and reviewed annually or semiannually cannot keep pace. Policy violations often occur faster than data governance teams can detect and respond to before they cause business problems. In addition, the use of AI <a href="https://www.techtarget.com/searchdatamanagement/opinion/AI-agents-push-enterprises-toward-unified-data-governance">creates new governance issues</a> that require more frequent policy updates.</p> <p>Instead of relying on retrospective audits, AI models and agents continuously monitor data use and identify policy violations as they occur. AI can analyze the use of large volumes of data across complex environments, ensuring governance coverage that would be impractical with manual processes. Machine learning models add contextual awareness, evaluating factors such as data sensitivity, user behavior and risk levels.</p> <p>These capabilities transform data governance into an always-on function that adapts more easily as conditions change. Governance can also be integrated directly into data workflows, providing continuous visibility <a href="https://www.techtarget.com/searchdatamanagement/tip/Experts-share-practices-to-overcome-AI-data-readiness">throughout the data lifecycle</a> and enabling organizations to identify issues early and prevent them from becoming significant business risks.</p> <p>As organizations use this increased visibility and monitoring to inform ongoing governance policy reviews, GenAI tools can also help by accelerating activities such as drafting new or updated policies, data standards, business definitions and data stewardship documentation.</p> </section> <section class="section main-article-chapter" data-menu-title="Core components of AI-powered policy enforcement"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Core components of AI-powered policy enforcement</h2> <p>The following foundational components work together to support AI-driven data governance.</p> <h3>Intelligent data classification</h3> <p>AI can automatically classify data based on rules established by data owners and stewards. It uses data sensitivity guidelines, regulatory requirements and business context to identify items such as personal information, financial records and intellectual property. Accurate data classification ensures AI tools consistently apply the right governance controls.</p> <h3>Policy as code</h3> <p>To enable automated monitoring and enforcement, data governance policies must be machine-readable. A policy-as-code framework translates governance rules, such as data access controls, retention requirements and encryption standards, into formats that AI systems can understand.</p> <h3>Real-time monitoring and anomaly detection</h3> <p>Continuously monitoring data use enables AI to identify anomalies, such as unusual access patterns or unexpected data transfers, in real time. As AI models learn the typical behavior of an organization's data users, their ability to detect subtle governance issues and risks will further improve.</p> <h3>Automated enforcement mechanisms</h3> <p>Once a policy violation is detected, AI can act autonomously to mask data, restrict access, require authentication or prevent misuse. Automated enforcement by AI reduces response time and limits the negative effects of <a href="https://www.techtarget.com/searchdatamanagement/feature/Top-3-data-privacy-challenges-and-how-to-address-them">data privacy and security problems</a>.</p> </section> <section class="section main-article-chapter" data-menu-title="Designing escalation paths for human oversight"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Designing escalation paths for human oversight</h2> <p>Despite AI's automation capabilities, human involvement in enforcing governance policies remains critical. Effective data governance requires judgment, context and accountability that AI alone cannot provide. Clear escalation paths ensure the right stakeholders review the AI's actions and any complex or high-risk issues it identifies.</p> <p>For example, human reviews, both scheduled and ad hoc, should complement real-time AI monitoring to validate its results. Similarly, human oversight of automated enforcement decisions ensures they're appropriate and aligned with an organization's governance policies.</p> <p>Escalation paths and AI oversight plans should include the following responsibilities for <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-roles-and-responsibilities-Whats-needed">key participants in data governance programs</a>:</p> <ul class="default-list"> <li><b>Data governance managers.</b> They lead the process of setting governance strategies, defining policies and <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more">measuring the effectiveness of governance initiatives</a>. As part of that, they should also take the lead in developing and implementing AI tools for governance functions. This includes ensuring AI models and agents are trained appropriately, monitoring their use and escalating governance issues to data owners and business stakeholders when necessary.</li> <li><b>Data owners.</b> They are accountable for AI-driven decisions on data classification and access controls, as well as ensuring AI recommendations and enforcement actions conform to governance policies and approved exceptions.</li> <li><b>Data stewards.</b> They help validate AI training datasets and outputs, refine data classifications, resolve ambiguous enforcement cases and improve model accuracy, supporting the work of data owners and data governance managers.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Best practices for adopting AI-driven data governance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Best practices for adopting AI-driven data governance</h2> <p>Organizations should take a focused, practical approach to incorporating AI into data governance, beginning with high-risk domains such as customer, financial or healthcare data, where <a href="https://www.techtarget.com/searchdatamanagement/opinion/The-future-of-AI-depends-on-better-data-not-bigger-models">strong governance controls and data stewardship</a> are most critical.</p> <p>These are additional best practices for data leaders looking to use AI to monitor data use and enforce governance policies:</p> <ul class="default-list"> <li>Establish clear guardrails defining what AI systems are authorized to generate, access, recommend or execute as part of data governance initiatives.</li> <li>Document automated governance processes, AI decision rules and human oversight workflows, and reinforce them through ongoing training of data governance teams.</li> <li>Treat AI-driven data governance as a continuous improvement process, using feedback from data stewards, owners and governance teams to refine rules, standards, workflows and automation prompts.</li> <li>Prioritize transparency, auditability and compliance to ensure AI-assisted governance decisions are traceable and trustworthy and meet legal and ethical requirements.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="The future of data governance with AI"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The future of data governance with AI</h2> <p>Data governance will continue to evolve as AI advances. Future AI-driven governance models will likely include adaptive policies and context-aware controls that adjust dynamically to changing datasets or data use.</p> <p>However, data governance isn't just a technical function -- and that won't change even as organizations increasingly rely on AI to drive governance efforts. The most effective approaches will continue to combine AI's speed and scalability with human expertise and decision-making.</p> <p>Successful governance depends on maintaining a balance: using AI to increase efficiency and automate policy enforcement, while ensuring that <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-responsibilities-now-belong-in-the-C-suite">data leaders and business stakeholders</a> provide judgment, accountability and strategic guidance.</p> <p><em>Anne Marie Smith, Ph.D., is an information management professional and consultant with broad experience across industries. She has also designed and delivered numerous data management courses and educational programs.</em></p> </section> AI models and agents can automate data governance policy enforcement for proactive governance in dynamic data environments. But don't forget the human element. https://cdn.ttgtmedia.com/visuals/searchEnterpriseAI/business-strategies/searchEnterpriseAI_036.jpg https://www.techtarget.com/searchdatamanagement/tip/How-to-use-AI-to-enforce-data-governance-policies Thu, 09 Jul 2026 10:24:00 GMT <p>A quiet anxiety is running through many AI conversations right now. Companies have spun up agents across departments and most aren't making it to production.</p> <p>The instinct is to read that as a failure, but it isn't.</p> <p>We've seen this pattern before -- during the mobile app boom -- and the organizations that understood it last time came out ahead. This time, the payoff is even bigger.</p> <section class="section main-article-chapter" data-menu-title="We've seen this movie before"> <h2 class="section-title"><i class="icon" data-icon="1"></i>We've seen this movie before</h2> <p>The AI agent boom has many similarities to the mobile app explosion. In 2025 alone, there were 4.2 million apps on Google Play and 2.5 million on the Apple App Store, with thousands of new apps shipping daily. But the overwhelming majority never found an audience.</p> <p>In 2015, Andrew Chen, a partner at Andreessen Horowitz, <a target="_blank" href="https://andrewchen.com/new-data-shows-why-losing-80-of-your-mobile-users-is-normal-and-that-the-best-apps-do-much-better/" rel="noopener">reported</a> the average app lost 77% of its users within three days, 90% within a month and 95% within 90 days. By any production-grade metric, almost all of them failed.</p> <p>But that framing missed the point. Most of those apps were never destined to be commercial hits. They were how companies learned mobile-first thinking, including:</p> <ul class="default-list"> <li>How to design for a small screen.</li> <li>How to earn attention.</li> <li>How to build for a new type of user.</li> </ul> <p>The "failures" were tuition. The companies that treated the experimental phase as education instead of demanding a hit from every release built the muscles that defined the next decade.</p> <p>AI agents are on the same journey. But this time the lesson cuts deeper, and it pays off in two ways at once.</p> </section> <section class="section main-article-chapter" data-menu-title="The difference that changes everything"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The difference that changes everything</h2> <p>One thing separates the two eras: A mobile app lets you defer the data reckoning. An agent forces it.</p> <p>A mobile app could succeed despite a messy backend. Clever design and <a href="https://www.techtarget.com/searchapparchitecture/tip/App-interface-design-principles-all-developers-should-know">a smooth interface</a> could paper over years of data chaos that the user never saw. That's partly why people happily keep 80 apps installed while actively using only nine. The polish was real, even when the substance was thin.</p> <p>An agent has nowhere to hide. It's <a href="https://www.techtarget.com/searchenterpriseai/opinion/The-critical-role-of-data-in-building-an-AI-agent">only as good as the data</a> it can reach, understand and act on. While building one, every weakness in your data surfaces with uncomfortable precision, including:</p> <ul class="default-list"> <li>Incompatible systems and inconsistent formats.</li> <li>Missing or meaningless metadata.</li> <li>A "source of truth" that three departments define differently.</li> <li>Institutional knowledge that lives in people's heads and was never written down.</li> <li>Integration debt between the CRM, the ticketing system, the product database and the knowledge base, which all suddenly must speak to each other.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="What's left behind is worth more than the agent"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What's left behind is worth more than the agent</h2> <p>A shelved agent isn't a wasted one. While the agent might not survive, the data work does.</p> <p>Every agent experiment forces an organization to complete tasks it kept putting off. By the end of an agent's development, organizations will have documented tribal knowledge, set data quality standards, <a href="https://www.techtarget.com/searchdatamanagement/definition/data-silo">broken down team silos</a>, defined ownership and governance and built the integration layers that make data usable everywhere. None of that disappears when the pilot ends. It compounds.</p> <p>In practice, this looks like the customer service agent that never launched leaving behind a unified, cleaned customer record that improves every system downstream. The discarded research assistant leaving a tagged, searchable knowledge base that employees use for years. Or the sales agent that stalled in testing and left standardized pipelines that power better analytics across the business.</p> </section> <section class="section main-article-chapter" data-menu-title="People quietly gain AI skills"> <h2 class="section-title"><i class="icon" data-icon="1"></i>People quietly gain AI skills</h2> <p>The part that does not show up on any dashboard and makes this such an exciting moment to be inside an organization is the people involved.</p> <p>In the mobile era, building was mostly left to specialists. This time, it's everywhere. People who have never written a line of production code -- marketers, analysts, operations leads, support managers, finance teams -- are prototyping agents, testing what AI can and cannot do, and building real intuition about where it helps. They are learning the grain of the technology by using it instead of sitting through a training deck.</p> <p>That's a staggering amount of capability spreading through a company at once, and you couldn't have mandated it. No budget line buys <a href="https://www.techtarget.com/searchenterpriseai/feature/Why-AI-literacy-is-critical-even-for-non-technical-employees">AI literacy</a> at this speed. The mobile boom taught a generation of teams to think about products and users. The agent boom teaches a far broader group of people to <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">think about data, automation, and what good judgment looks like</a> when a machine is doing the work. Human learning compounds alongside the data, and it may end up being the more valuable of the two.</p> </section> <section class="section main-article-chapter" data-menu-title="You are counting the wrong thing"> <h2 class="section-title"><i class="icon" data-icon="1"></i>You are counting the wrong thing</h2> <p>The metric to watch isn't the number of agents in production. This early in the market, that number tells you almost nothing.</p> <p>The metrics that matter are quieter: how much cleaner and <a href="https://www.techtarget.com/searchdatabackup/tip/Enterprise-data-governance-Frameworks-and-best-practices">better governed</a> your data is than it was a year ago, and how many people in the building know how to put AI to work. The organizations that will own the AI-native era are using this phase to understand their data landscape, raise their data quality and turn thousands of curious employees into confident builders.</p> </section> <section class="section main-article-chapter" data-menu-title="Data-first thinking is key"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Data-first thinking is key</h2> <p>The mobile revolution taught us to think mobile-first, but when a failed app vanished from the store, it took its lessons with it. However, the agent boom teaches us to think data-first, as the data work behind an agent gets reused, making the next ten agents faster, while the people who built it keep the skills for good.</p> <p>So, when the next agent pilot gets shelved, log it as tuition paid toward becoming a data-ready, AI-literate company. After all, winners in the mobile era weren't the apps with the most downloads, but those who learned the most from them. The same is true in the agent era -- the winners will be those whose data and people are ready when it counts.</p> <p><em>Stephen Catanzano is a senior analyst at Omdia, where he covers data management and analytics.</em></p> <p><em>Omdia is a division of&nbsp;Informa TechTarget.&nbsp;Its analysts have business relationships with technology vendors.</em></p> </section> Most agent pilots will never reach production, but these failures can prove useful to data operations. Organizations should look to the 2010s mobile app boom for guidance. https://cdn.ttgtmedia.com/visuals/digdeeper/4.jpg https://www.techtarget.com/searchdatamanagement/opinion/AI-agent-failures-offer-hidden-value-for-enterprises Wed, 01 Jul 2026 15:30:00 GMT <p><i>Data platform vendors are lining up behind Apache Iceberg, with several recently announcing new or expanded support for the open table format used to manage large analytics datasets in data lakes and lakehouses.</i></p> <p><i>For example, Snowflake made <a href="https://www.techtarget.com/searchdatamanagement/news/366641473/Snowflake-broadens-open-source-embrace-ups-Iceberg-support">support for Version 3 of the Iceberg specification</a> generally available in May, a month after enabling users to store Iceberg tables in its platform. These moves further deepen the full embrace of the table format that Snowflake announced in April 2025, after previously offering only limited support for Iceberg.</i></p> <p><i>Also in May, Databricks added a broad set of Iceberg capabilities to its data lakehouse platform, including Iceberg V3 support and the ability to create Iceberg tables in its Unity Catalog software. Its support, which began with the 2024 acquisition of a startup <a href="https://www.techtarget.com/searchdatamanagement/news/366588032/Databricks-1B-plus-Tabular-acquisition-adds-Iceberg-support">whose founders included Iceberg's creators</a>, is particularly notable because Databricks initially developed the alternative Delta Lake table format.</i></p> <p><i>SAP is also going the acquisition route: In May, it said it's <a href="https://www.techtarget.com/searchdatamanagement/news/366642794/SAP-acquisitions-of-Dremio-Prior-Labs-target-AI-development">buying Iceberg-based data lakehouse vendor Dremio</a>. The deal will extend the reach of SAP's Business Data Cloud to new external data sources without forcing users to move data into the SAP platform.</i></p> <p><i>In a Q&amp;A with TechTarget, Donald Farmer, principal of consulting firm TreeHive Strategy, said these and various other vendor announcements demonstrate that Iceberg has become the de facto industry-standard table format, beating out both Delta Lake and Apache Hudi. Vendors really have no choice but to support it, he added.</i></p> <p><i>Farmer also discussed Iceberg's capabilities and issues that data leaders and teams need to consider when planning deployments of the table format.</i></p> <p><b>Editor's note:</b>&nbsp;<i>This Q&amp;A has been edited for clarity and conciseness</i>.</p> <p><b>Why are data platform vendors rushing to add or expand support for Iceberg?</b></p> <p>Donald Farmer: Partly, it's because the question of table format is now settled. Iceberg has won, and it is now the default. Once that happens, as a vendor, you can't sit outside that process. You have to adopt it.</p> <div class="imagecaption alignLeft"> <img src="https://cdn.ttgtmedia.com/rms/onlineImages/farmer_donald.jpg" alt="Photo of Donald Farmer, principal of TreeHive Strategy.">Donald Farmer </div> <p>The first thing is wanting to defend against being excluded [by technology buyers]. You always want to be in on the RFP, so if Iceberg is in the RFP, you need to support it. Also, no matter how big you are as a vendor, once something becomes commoditized in that way, you have to support it. Look at SAP, which acquired Dremio. They didn't have an Iceberg-native engine. They had to find one.</p> <p>If you think of that as a negative framing -- "I don't want to be excluded" -- the positive framing is that the <a href="https://www.techtarget.com/searchdatamanagement/opinion/Why-Apache-Iceberg-is-essential-for-modern-data-lakehouses">Iceberg-native data lakehouse</a> is now the substrate for AI agents. One of the problems with agentic AI is that people see the data architecture that underlies the agents as <a href="https://www.techtarget.com/searchdatamanagement/opinion/AI-agents-push-enterprises-toward-unified-data-governance">being fragmented and siloed</a> in their organizations. The Iceberg-native lakehouse pools all that together and provides a fairly neutral environment for agents to run over, rather than trying to build agents over this fragmented, siloed architecture.</p> <p><b>For data leaders and teams, what capabilities does Iceberg provide that aren't supported by what could perhaps be called traditional data lakes at this point?</b></p> <p>Farmer: The traditional data lake is more or less just files in an object store -- particularly, more recently, Parquet files in an object store, with a Hive-style directory. Iceberg adds a metadata tier over that. It's not just file storage; there's the metadata tier, and also a sort of transaction layer. To a certain extent, you can have ACID transactions, for example.</p> <p>Iceberg also supports schema evolution, so you can add columns, you can rename them, maybe even change the type of a column, but you don't have to rewrite the data file back to that. And it has similar support for partition evolution, so you can change the partitioning schema without rewriting the existing data. It has capabilities like that that go above what you could also call the kind of dumb data lake, which is just files in an object store with a loose structure around them.</p> <p><b>Why has Iceberg become the default table format instead of Delta Lake or Hudi, or the three being more equal competitors?</b></p> <p>Farmer: To be fair, Delta still has a very large installed base. It's the native default format in <a href="https://www.techtarget.com/searchdatamanagement/news/366643955/Microsoft-boosts-Fabric-to-make-it-a-foundation-for-AI">Microsoft Fabric</a>, for example. It's not as if Delta has gone away; it's just not the open industry standard.</p> <p>Part of this is governance. Iceberg is part of the Apache Software Foundation, while Delta came from Databricks -- and even though it's open source, you still get the impression that there's single-vendor control. I think that's an issue -- people are very averse to being locked into any technologies. Also, if you look at the catalog layer, Iceberg has a REST catalog API, while Delta is cataloged through Databricks's Unity Catalog, which is powerful but still vendor-specific.</p> <p>As for Hudi, it's really good in certain scenarios, like high-volume, high-frequency streaming or change data capture. It's got really good record-level indexing that enables that, and a merge-on-read system that enables it to keep data current with very high performance for streaming scenarios.</p> <p>That sounds like a mixed story, but it really isn't. Iceberg is the standard. It's dominant.</p> <p><b>Apache XTable is an incubating technology that supports interoperability between the three table formats. Delta Lake also has a Universal Format, or UniForm, feature that lets users read its tables in Iceberg and Hudi. Do you expect to see many mixed environments with the different table formats?</b></p> <p>Farmer: I expect we are going to see convergence. Version 3 of Iceberg has done a pretty good job of unifying the data layer. It's got features like row lineage, for example, and the Variant data type, which is important for storing semistructured data. And Databricks has said pretty clearly that eventually, Delta and Iceberg will just use the same metadata and share tables.</p> <p>Yeah, there are some bridges like UniForm and XTable, which has good backing -- Microsoft and Google are backing it. But I think the pattern [of sharing metadata and tables] is probably the way things are going, rather than having a lasting split between the three different formats with integrations between them. Iceberg becomes the lingua franca of data.</p> <p><b>Does Iceberg have any limitations or challenges that data teams looking to deploy it should be aware of? Migration costs, for example?</b></p> <p>Farmer: There is a migration cost, and if you are doing it at scale -- migrating petabytes, which is absolutely possible nowadays -- that's a huge migration. And there are some things that people get stuck on -- issues with partitioning and file paths, for example, that make it messy to migrate. But even without that messiness, it <a href="https://www.techtarget.com/searchdatamanagement/opinion/Is-Apache-Iceberg-worth-a-full-migration">can just be a big job</a>.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> You have to build maintenance into your operational plan for Iceberg. Too many people discover that only after the system has started to degrade. </figure> <figcaption> <strong>Donald Farmer</strong>Principal, TreeHive Strategy </figcaption> <i class="icon" data-icon="z"></i> </div> </blockquote> <p>I think maintenance is a bigger issue. The way Iceberg works is it writes new metadata and new data files for every change. That creates the potential for performance deterioration over time. You end up with something a little bit like the old days when you had to defragment your disk drive. You have this proliferation of very small files, and you need to schedule maintenance of that. You've got to do file compaction, and you have to do expiration of data snapshots that have been taken. You need to clean all that out.</p> <p>As a result, you have to build maintenance into your operational plan for Iceberg. Too many people discover that only after the system has started to degrade. No doubt this will get fixed over time. But right now, there is a maintenance cost for Iceberg that there is not for Delta. That, I think, is one of the reasons we haven't seen a wholesale migration from Delta to Iceberg.</p> <p><b>Any additional advice or best practices for data teams on deploying and managing Iceberg?</b></p> <p>Farmer: My advice to people who ask me about these formats is that it's not just one decision. There's the choice of the table format, there's the catalog, there's a query engine, there's the governance and maintenance you have to do. Iceberg is a pretty straightforward choice now. But the catalog, in particular, is a sticky decision that you need to get right.</p> <p>Iceberg's REST catalog spec is very portable and has a great API. You could use Apache Polaris, which is an open source catalog purpose-built for Iceberg. But you have other options, which could include Unity Catalog, Snowflake Horizon Catalog, Dremio Open Catalog, AWS Glue, Hive Metastore and other catalogs. It's a really important decision. The choice of a catalog is going to define how Iceberg integrates into your enterprise environment.</p> <p><em>Craig Stedman is an industry editor at TechTarget who edits and writes articles on data technologies and processes. He has covered enterprise IT for more than 40 years.</em></p> In a Q&A, consultant Donald Farmer explains why vendors are rushing to support Apache Iceberg and discusses its capabilities and deployment issues for data teams. https://cdn.ttgtmedia.com/visuals/searchDataManagement/data_warehouse/datamanagement_article_006.jpg https://www.techtarget.com/searchdatamanagement/feature/Why-Apache-Iceberg-is-the-center-of-attention-in-data-platforms Tue, 30 Jun 2026 22:19:00 GMT <p>Effective data integration is essential for generating accurate analytics insights and AI outcomes in big data applications. But big data integration requires a shift away from traditional integration techniques to handle large volumes of diverse data often collected and processed at high velocity.</p> <p><a href="https://www.techtarget.com/searchdatamanagement/The-ultimate-guide-to-big-data-for-businesses">Big data environments</a> provide new opportunities to derive insights from unstructured and semistructured data, such as website and application logs, emails, social media posts, images and IoT data streams. Conventional integration approaches fall short when teams need to work with this data, said Rosaria Silipo, a data scientist, author and co-host of the "My Data Guest" podcast.</p> <p>Data integration is especially challenging when volume, variety and velocity -- the core 3 V's of big data -- are all factors in analytics and AI applications. Ad hoc integration for individual projects isn't viable in such scenarios, Silipo said. To avoid problems and <a href="https://www.techtarget.com/searchbusinessanalytics/feature/6-big-data-benefits-for-businesses">maximize business value</a>, data leaders must develop a comprehensive integration strategy that addresses big data's scale and complexity.</p> <section class="section main-article-chapter" data-menu-title="New challenges, new approaches"> <h2 class="section-title"><i class="icon" data-icon="1"></i>New challenges, new approaches</h2> <p>Current and complete data is <a href="https://www.techtarget.com/searchdatamanagement/feature/Effective-integration-key-to-creating-trusted-data">critical to delivering trustworthy insights</a> for business decision-making. But the extract, transform and load (ETL) integration approach used in traditional data warehouses is often a nonstarter in big data systems, said David Mariani, co-founder and CTO of semantic layer platform vendor AtScale.</p> <p>Batch ETL processes struggle with large data volumes due to data transformation bottlenecks, making it difficult to keep pace with frequent data updates and dynamic analytics requirements. For example, an online retailer's overnight ETL jobs processing millions of daily transactions might exceed their processing windows, leaving business executives and analysts without up-to-date sales data the next morning. ETL is also poorly suited to unstructured or semistructured data because it requires data to be transformed into a predefined schema before loading.</p> <p>The alternative ELT approach addresses these limitations by reversing the load and transform steps. Data is loaded into a data lake or lakehouse in its native format, then transformed and integrated as needed for specific <a href="https://www.techtarget.com/searchbusinessanalytics/feature/8-big-data-use-cases-for-businesses-and-industry-examples">big data use cases</a>. ELT increases scalability, enabling data teams to process large volumes more efficiently and handle high-velocity updates. It also provides greater flexibility for supporting new applications and <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-identify-and-manage-AI-model-drift">updating AI and analytics models</a> as data changes.</p> <p>Many organizations also deploy real-time data integration and processing technologies to deliver immediate insights in time-sensitive applications such as fraud detection, real-time personalization and operational or patient monitoring. Common real-time technologies include stream processing, event-driven architecture and change data capture. They enable teams to capture data as it's generated or updated and continuously load it into data platforms, often using an ELT approach to structure the data for different applications.</p> </section> <section class="section main-article-chapter" data-menu-title="How AI agents complicate big data integration"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How AI agents complicate big data integration</h2> <p>The <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-prepare-your-business-for-agentic-AI-adoption">rise of agentic AI</a> further complicates big data integration. Basic integration involves a one-directional pipeline: Data flows from source systems into a repository for analysis. More advanced applications support bidirectional data integration that directly feeds analytics insights back into operational systems. As organizations increasingly <a href="https://www.techtarget.com/searchcio/feature/Agentic-ai-in-practice-lessons-from-real-deployments">deploy AI agents</a>, data teams might need to implement bidirectional integration at a much larger scale.</p> <p>AI agents don't just access and analyze the data in a data lake or lakehouse, said David DuChene, senior manager of data and AI professional services at SHI International. They generate new outputs and surface latent relationships across data domains. Agents also autonomously push enriched insights back to the original source systems if configured to do so. That capability requires more extensive bidirectional integration capabilities, often operating continuously, DuChene said.</p> <p>Data governance pressures also <a href="https://www.techtarget.com/searchdatamanagement/feature/How-agentic-AI-governance-tackles-data-security-challenges">increase with agentic AI</a>. Dan Federoff, vice president and head of data solutions at IT consultancy Bridgenext, said agents don't push back on bad data; they stealthily spread it across an organization. That makes effective data governance -- including strong data quality management and <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">comprehensive data lineage documentation</a> -- even more critical to successful big data integration in the AI era, Federoff said.</p> <p>In addition, authorization and access control must become more dynamic, said Steve Touw, co-founder and CTO at data security platform vendor Immuta. AI agents operate at machine speed, often across systems and on behalf of multiple users with different permission levels. Touw recommended <a href="https://www.techtarget.com/searchsecurity/tip/Cybersecuritys-agentic-AI-identity-crisis-and-how-to-fix-it">assigning identities to agents</a> that dynamically assume the permissions of different users and configuring the integration layer to provision ephemeral, just-in-time roles. Doing so enables an agent to query data without holding permanent access privileges.</p> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineimages/data_management-data_integration.png"> <img data-src="https://www.techtarget.com/rms/onlineimages/data_management-data_integration_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineimages/data_management-data_integration_mobile.png 960w,https://www.techtarget.com/rms/onlineimages/data_management-data_integration.png 1280w" alt="Visual that lists common data integration techniques and methods." height="359" width="560"> <figcaption> <i class="icon pictures" data-icon="z"></i>ELT, change data capture and streaming data integration are commonly used in big data environments. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="Best practices for big data integration"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Best practices for big data integration</h2> <p>Adopt the following best practices to smooth out the big data integration process and ensure it meets business needs.</p> <h3>Create an all-encompassing data integration strategy</h3> <p>Rick Skriletz, founder and CEO of IT services provider InfoNovus Technologies, said successful integration efforts must work in harmony with several related data management functions:</p> <ul class="default-list"> <li>Data collection across multiple systems.</li> <li>Data processing, storage, security and preparation.</li> <li>Data backup for disaster recovery.</li> </ul> <p>Without a cohesive integration strategy that clearly details how these functions all fit together, data management teams are likely to address each of them separately -- and less effectively, Skriletz said.</p> <h3>Treat data as a product</h3> <p>Data is often viewed as a byproduct of applications and systems. However, a cultural shift toward <a href="https://www.techtarget.com/searchbusinessanalytics/opinion/The-importance-of-data-products">treating data as a product</a> in its own right helps make the integration process more effective, said Jeffrey Pollock, vice president of data replication and streaming products at Oracle. That involves applying product management principles to data assets, with clear ownership, a defined purpose and a focus on data quality, usability and reliability.</p> <p>Netflix has bought into the concept. Tomasz Magdanski, senior manager of data science and engineering at the streaming company, wrote in an October 2025 <a target="_blank" href="https://netflixtechblog.medium.com/data-as-a-product-applying-a-product-mindset-to-data-at-netflix-4a4d1287a31d" rel="noopener">blog post</a> that it's adopting a data-as-a-product framework "to ensure our data assets are managed with the same rigor and strategic focus as traditional products." The goal, he said, is to "elevate data to a first-class entity in our organization's thinking" and ensure that it's trustworthy and aligned with strategic business goals.</p> <h3>Capture critical metadata</h3> <p>Data integration depends on metadata that describes the data elements flowing into a data lake or lakehouse. It's commonly captured in two tools for different users:</p> <ul class="default-list"> <li>A logical data model documents business and technical metadata that data teams use to plan integration work and resolve common problems -- such as data with different structures, inconsistent naming conventions and varying quality levels.</li> <li>A data catalog creates an <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">inventory of data assets with associated metadata</a>, enabling data scientists and business users to find relevant data, understand its context and meaning, and identify datasets that need to be integrated for new use cases.</li> </ul> <h3>Incorporate integration into a data lifecycle management framework</h3> <p>Data lifecycle management (<a href="https://www.techtarget.com/searchstorage/definition/data-life-cycle-management">DLM</a>) establishes policies and procedures for managing data from its creation through archiving and deletion. Data integration should be a core component of an organization's DLM framework. Incorporating it into a systematic framework -- with documented methodologies and governance structures -- is especially critical in big data environments, where volume and complexity make a less rigorous approach unsustainable.</p> <h3>Take an enterprise-wide approach</h3> <p>Maximizing the business value of big data initiatives requires an enterprise-wide approach to data integration rather than isolated implementations in individual departments or business units,&nbsp;according to Faisal Alam, technology consulting leader for the industrial and energy sectors at EY Americas. Siloed integration efforts limit the cross-functional data access and analytics insights needed for strategic decision-making across an organization.</p> <p><b>Editor's note:</b><i> This article was updated in June 2026 for timeliness and to add new information.</i></p> <p><em>George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.</em></p> </section> Data integration in big data systems is even more complex now because of AI. To succeed, it requires a strategy built on new approaches and strong data management. https://cdn.ttgtmedia.com/visuals/searchDataManagement/data_warehouse/datamanagement_article_006.jpg https://www.techtarget.com/searchdatamanagement/feature/Establish-big-data-integration-techniques-and-best-practices Thu, 04 Jun 2026 14:53:00 GMT <p>As AI moves from contained pilots into production and agentic systems, enterprises have learned a hard lesson: the model is rarely the problem. Meaning is, and the missing ingredient is context.</p> <p>Most enterprises have plenty of data. What they lack is a shared, governed layer of meaning that travels with that data wherever it lives. That <a href="https://www.techtarget.com/searchenterpriseai/tip/Exploring-the-context-layer-for-AI-systems">business context layer</a> becomes the foundation for everything else.</p> <section class="section main-article-chapter" data-menu-title="Why context matters"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why context matters</h2> <p>Context is the business meaning wrapped around raw data: the definitions, relationships, rules and lineage that tell a model what a column, a metric, or an entity signifies. An AI system that does not understand what data represents produces fast, fluent and confidently wrong answers at a production scale that <a href="https://www.techtarget.com/searchenterpriseai/feature/How-to-ensure-AI-transparency-explainability-and-trust">erodes trust</a> faster than any accuracy benchmark can rebuild.</p> <p>Context is the difference between a number that signifies customers and a number that signifies quarterly net revenue under the current recognition policy. Without it, "churn" means one thing to finance, another to product and a third to sales. An agent asked about churn will blend those contradictions into outputs no one can act on.</p> </section> <section class="section main-article-chapter" data-menu-title="Context expands scope"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Context expands scope</h2> <p>Context used to sit quietly in a catalog that analysts consulted by hand. Now it must do the following:</p> <ul class="default-list"> <li>Be delivered at runtime to queries, models and autonomous agents acting on data without a <a href="https://www.techtarget.com/searchenterpriseai/feature/Humans-and-AI-The-role-of-people-in-the-new-AI-world">human in the loop</a>.</li> <li>Span fragmented environments across clouds, lakes, SaaS applications and operational systems.</li> <li>Stay governed. An agent that can open tickets, update records and trigger workflows is only as safe as its inherited business rules and guardrails.</li> </ul> <p>An example of a company building for this reality is Starburst. Rather than forcing organizations to centralize data before AI can use it, its Enterprise Intelligence Platform brings AI to the data. Its AI-ready data products combine governed data, metadata and business definitions into reusable, trusted assets that provide consistent business context at runtime regardless of where data resides. Starburst's <a href="https://www.techtarget.com/searchdatamanagement/news/366643641/New-Starburst-platform-extends-AI-to-distributed-data">AI Data Assistant</a> then delivers that context where business users work -- inside applications, workflows and agents -- so meaning follows the question instead of rebuilding it for every model.</p> <p>But this strategic point is bigger than any one vendor. Context, not compute, is becoming the gating factor for production AI. Organizations continuing to layer models on fragmented, undefined data will keep receiving fast answers they cannot trust, while those investing in governed, distributed context will scale AI with confidence.</p> <p><em>Stephen Catanzano is a senior analyst at Omdia, where he covers data management and analytics.</em></p> <p><em>Omdia is a division of&nbsp;Informa TechTarget.&nbsp;Its analysts have business relationships with technology vendors.</em></p> </section> Your AI model might not be the reason it fails in production. Missing or unclear context throws off the model, leading to untrustworthy outputs that the business can't act on. https://cdn.ttgtmedia.com/visuals/digdeeper/4.jpg https://www.techtarget.com/searchdatamanagement/opinion/Context-is-the-make-or-break-layer-for-AI-in-production Thu, 04 Jun 2026 10:58:00 GMT <p>Most data problems do not announce themselves.</p> <p>A pipeline runs, rows of data arrive, dashboards refresh -- but somewhere along the line, a feature quietly starts behaving differently than it did last week. By the time anyone notices, an AI model has been making faulty predictions for days.</p> <p>Data observability catches these silent failures by applying site reliability engineering principles to data: instrumentation, monitoring and alerting.</p> <section class="section main-article-chapter" data-menu-title="What is data observability?"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What is data observability?</h2> <p>Data observability is the continuous, automated monitoring of data's health <a href="https://www.techtarget.com/searchdatamanagement/tip/Top-data-observability-use-cases">as it moves through pipelines</a> and into models. Rather than waiting for downstream users to spot problems, observability tools watch the data and compare today's arrivals with recent history -- or with what the schema and data contract specify should arrive. When something is off, they raise an alert close to the source, ideally before bad data reaches a model or a user.</p> <p>The discipline is often described in five pillars:</p> <ul class="default-list"> <li><b>Freshness.</b> Is the data up to date?</li> <li><b>Volume. </b>Are row counts within the expected range?</li> <li><b>Schema</b>. Has the data's structure changed?</li> <li><b>Distribution. </b>Do the data values fall within expected boundaries?</li> <li><b>Lineage.</b> What upstream and downstream assets are affected by data issues?</li> </ul> <p>Together, they answer the basic question: Is this data still trustworthy?</p> </section> <section class="section main-article-chapter" data-menu-title="Why AI projects need observability more than traditional analytics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why AI projects need observability more than traditional analytics</h2> <p>A broken dashboard is annoying. A broken feature pipeline is dangerous. Machine learning (ML) models do not throw errors when their input data shifts. They keep producing predictions, <a href="https://www.techtarget.com/searchdatamanagement/tip/Experts-share-practices-to-overcome-AI-data-readiness">just inaccurate ones</a>. A categorical feature that suddenly has a new value, a numeric column whose mean has crept upward, a timestamp column with growing nulls: Any of these can silently degrade a model for weeks.</p> <p>Observability is foundational for detecting concept drift and <a href="https://www.techtarget.com/searchitoperations/feature/Meeting-the-challenges-of-scaling-AI-with-MLOps">training-serving skew</a>. Comparing live feature distributions against the training set -- automatically and continuously -- provides reliable signals that an ML model still operates in the world it was trained for. Without observability, teams must rely on lagging business metrics that reveal problems only after they've caused damage.</p> </section> <section class="section main-article-chapter" data-menu-title="What good data observability captures"> <h2 class="section-title"><i class="icon" data-icon="1"></i>What good data observability captures</h2> <p>A mature data observability practice does more than uptime checks. Effective systems also:</p> <ul type="disc" class="default-list"> <li>Trigger alerts about stalled upstream jobs or other data freshness issues in minutes, not days.</li> <li>Track expected volume ranges, accounting for daily and seasonal patterns.</li> <li>Monitor for added columns, type changes, new enum values and other schema changes.</li> <li>Compare live data against historical baselines to check value distributions.</li> <li>Measure<b> </b>null and uniqueness rates &nbsp;to catch shifts that often precede failures.</li> <li>Use statistical tests tuned for each model's inputs to identify feature-level drift.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="How to start with data observability"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How to start with data observability</h2> <p>Begin with the data assets your models depend on most directly: training tables, feature views and inference inputs.</p> <p>Add basic freshness and volume checks first, since they catch the most incidents with the least effort.</p> <p>Layer in schema and distribution monitoring as you learn what normal looks like.</p> <p>Use tools to <a href="https://www.techtarget.com/searchdatamanagement/feature/4-trends-that-will-shape-data-management-and-AI-in-2026">streamline the process</a> -- whether it's specialized technologies such as Monte Carlo, Soda and Great Expectations or the observability features built into modern data platforms and catalogs. The principle is the same regardless of tooling: Monitor near the data source, alert the producing team when problems are detected and route incidents to a data owner who can fix the issue.</p> <p>But avoid alert fatigue from the start. Tune thresholds against real history rather than guesses. Treat noisy checks like noisy production alerts: Fix them or remove them.</p> </section> <section class="section main-article-chapter" data-menu-title="How observability pays off by catching drift"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How observability pays off by catching drift</h2> <p>Data observability turns silent failures into loud ones and shifts incident response from forensic to preventive. Combined with data contracts and lineage, it <a href="https://www.techtarget.com/searchdatamanagement/opinion/Tracing-data-lineage-in-AI-systems">closes the loop</a>: Contracts define what data should be, lineage shows how it flows, and observability confirms that reality matches the agreement. For AI projects, where models can degrade invisibly and expensively, that <a target="_blank" href="https://www.gartner.com/en/newsroom/press-releases/2026-05-12-gartner-predicts-40-percent-of-organizations-deploying-ai-will-use-ai-observability-to-monitor-model-performance-by-2028" rel="noopener">verification</a> is key to making production trustworthy.</p> <p><i>Stephen Catanzano is a senior analyst at Omdia, where he covers data management and analytics.</i></p> <p><i>Omdia is a division of Informa TechTarget. Its analysts have business relationships with technology vendors.</i></p> </section> Data quality issues get amplified in AI applications. Models can produce confident -- but misleading -- forecasts and conclusions without observability safeguards. https://cdn.ttgtmedia.com/visuals/digdeeper/1.jpg https://www.techtarget.com/searchdatamanagement/opinion/Data-observability-for-AI-helps-curb-poor-model-performance Tue, 26 May 2026 13:47:00 GMT <p>A well-executed big data strategy helps enterprises improve operational performance, optimize marketing campaigns and prioritize product development plans. But data leaders face various challenges in advancing <a href="https://www.techtarget.com/searchdatamanagement/The-ultimate-guide-to-big-data-for-businesses">big data initiatives</a> from boardroom discussions to successful deployments.</p> <p>Data teams must work with IT to build an infrastructure that collects diverse data from numerous sources and makes it available for use in analytics and AI applications. They also need to ensure big data systems meet performance, scalability and timeliness requirements, with high data quality and strong <a href="https://www.techtarget.com/searchdatamanagement/definition/data-governance">data governance</a> controls -- while also controlling implementation costs.</p> <p>Perhaps most importantly, data leaders must engage with business executives to determine&nbsp;<a href="https://www.techtarget.com/searchbusinessanalytics/feature/6-big-data-benefits-for-businesses">how big data can benefit the organization</a>&nbsp;and align the strategy with key business goals and priorities.</p> <p>Looking more deeply at these issues, here are 10 common big data challenges, along with advice on overcoming them.</p> <section class="section main-article-chapter" data-menu-title="1. Managing large volumes of data"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Managing large volumes of data</h2> <p>Big data typically involves large volumes of data from disparate systems, applications and external sources. It also usually includes a mix of structured, unstructured and semistructured data, which is often created or updated at a fast pace. Managing this combination of volume, variety and velocity -- the traditional 3 V's of big data -- is inherently complicated.</p> <p>That starts with extracting and consolidating relevant data from all the different sources -- CRM and ERP systems, website and application logs, sensors, social networks and more -- into a <a href="https://www.techtarget.com/searchdatamanagement/feature/Building-a-big-data-architecture-Core-components-best-practices">unified big data architecture</a>. Such architectures commonly have been built on data lakes, scalable platforms that store diverse types of data. But Donald Farmer, principal at consulting firm TreeHive Strategy, wrote in a TechTarget article on <a href="https://www.techtarget.com/searchbusinessanalytics/tip/6-essential-big-data-best-practices-for-businesses">big data best practices</a> that many data lakes are more like swamps, with sprawling data sets that are difficult to track and manage effectively.</p> <p>Farmer added that <a href="https://www.techtarget.com/searchdatamanagement/news/366545117/Lakehouse-architecture-the-best-fit-for-modern-data-needs">newer data lakehouse platforms</a> help ease those issues by combining the scalability and storage flexibility of data lakes with the more rigorous data management functions of traditional data warehouses. For example, he said Apache Iceberg and other open table formats provide transactional consistency and data versioning in data lakehouses, enabling data management teams to maintain audit trails and modify schemas without disrupting analytics and AI applications.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Finding and fixing data quality issues"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Finding and fixing data quality issues</h2> <p><a href="https://www.techtarget.com/searchenterpriseai/Ultimate-guide-to-artificial-intelligence-in-the-enterprise"></a>Big data applications produce bad results when data quality issues affect systems. These issues become more significant -- and harder to address -- as data management and analytics teams ingest more and more data. Monitoring data quality, identifying problems and fixing them is a continuous process, Bunddler CEO Paul Kovalenko said.</p> <p>Bunddler, an online marketplace for finding shopping assistants who help people buy products and arrange international shipments, experienced that firsthand as it scaled to 500,000 customers. The New York-based company uses big data to provide a highly personalized UX, monitor trends and identify upselling opportunities for assistants, but <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-quality-for-big-data-Why-its-a-must-and-how-to-improve-it">effective data quality management</a> is a pressing concern.</p> <p>Duplicate entries and typos are common in the data Bunddler collects from various sources, Kovalenko said. To root out such problems, it created a tool that matches duplicates with minor data differences and flags potential typos. Higher-quality data from using the tool has increased the accuracy of analytics insights, he said.</p> <p>AI can also help organizations improve data quality: It's increasingly being used to validate data and detect anomalies, errors, inconsistencies and other quality issues.</p> </section> <section class="section main-article-chapter" data-menu-title="3. Dealing with data integration complexities"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Dealing with data integration complexities</h2> <p>While big data platforms enable organizations to collect and store large amounts of varied data, the&nbsp;<a href="https://www.techtarget.com/searchdatamanagement/feature/Big-data-collection-processes-challenges-and-best-practices">data collection process</a>&nbsp;is challenging, said Rosaria Silipo, a data scientist, author and co-host of the "My Data Guest" podcast. In particular, integrating sets of big data is more complex than conventional data integration due to the different types of data involved and the fast pace of updates.</p> <p>Data leaders and teams need to think through their organization's data integration requirements upfront. Ad hoc integration for specific projects often results in redundant efforts and substantial rework of integration scripts or routines, Silipo said. Optimizing the ROI of big data investments requires a <a href="https://www.techtarget.com/searchdatamanagement/feature/Establish-big-data-integration-techniques-and-best-practices">strategic approach to data integration</a>, she added.</p> <p>That typically involves extract, load and transform (ELT) processes rather than the traditional ETL ones used in data warehouses. ELT loads data into a data lake or lakehouse in its native format, then combines and transforms it as needed for specific use cases. Real-time integration is also common in big data environments, and the growing adoption of AI tools and agents is accelerating a shift from rigid data pipelines to flexible architectures that <a href="https://www.techtarget.com/searchbusinessanalytics/news/365534255/Data-pipelines-deliver-the-fuel-for-data-science-analytics">deliver data to applications more dynamically</a>.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Scaling big data systems efficiently and cost-effectively"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Scaling big data systems efficiently and cost-effectively</h2> <p>Enterprises waste a lot of money collecting and storing big data if they don't have scalable systems capable of handling both current and future processing workloads. As a result, data teams should map out planned uses and required data types and schemas before designing and deploying big data systems.</p> <p>But that's easier said than done, said Travis Rehl, CTO and head of product at data, AI and cloud services provider Innovative Solutions. "Oftentimes, you start from one data model and expand out, but quickly realize the model doesn't fit your new data points -- and you suddenly have technical debt you need to resolve," Rehl said.</p> <p>Appropriate data structures make it easier to reuse data efficiently. For example, Parquet files often provide a better performance-to-cost ratio than CSV dumps within a data lake or lakehouse. Consistent <a href="https://www.techtarget.com/searchstorage/tip/Data-retention-and-destruction-policy-template-A-free-download">retention policies</a> cycle out old data from repositories as its analytics value erodes. When latency is an issue, teams also need to consider whether to run systems in the cloud, in on-premises data centers or <a href="https://www.techtarget.com/searchdatacenter/tip/From-core-to-edge-Strategies-for-scalable-compliant-and-agile-IT">on edge servers</a>, while balancing performance with deployment and management costs.</p> </section> <section class="section main-article-chapter" data-menu-title="5. Evaluating and selecting big data technologies"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Evaluating and selecting big data technologies</h2> <p>Data leaders and their teams can choose from a <a href="https://www.techtarget.com/searchdatamanagement/feature/15-big-data-tools-and-technologies-to-know-about">wide range of big data technologies</a> that often overlap in capabilities. Both open source tools and commercial platforms are available, further complicating the evaluation and selection process. Making the right choices is critical to gaining the expected business benefits from big data initiatives.</p> <p>To help inform technology decisions, teams should consider current and future data needs for both batch processing and real-time streaming from different sources. The <a href="https://www.techtarget.com/searchbusinessanalytics/definition/data-preparation">data preparation</a> capabilities required to support AI, machine learning and other advanced analytics applications should also be assessed, as well as where data will be processed and stored. The ability to easily update analytics and AI models in data platforms is another key consideration.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Generating valuable business insights"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Generating valuable business insights</h2> <p>The volume and complexity of big data complicate efforts to analyze and use it. Organizations often struggle to generate valuable insights and apply them in business operations in an impactful way, said Bill Szybillo, manager of BI engineering at firearms maker Sig Sauer Inc.</p> <p>Doing so requires a clear understanding of the data's business context and <a href="https://www.techtarget.com/searchbusinessanalytics/feature/8-big-data-use-cases-for-businesses-and-industry-examples">potential use cases</a>. But Silipo said she has found that many data leaders and teams focus on the technology and pay less attention to how big data systems can be used to achieve desired business outcomes.</p> <p>Teams that don't work with the people closest to business problems when planning data platforms, pipelines and storage architectures might build technically sound systems that produce little business value. Pilot projects are useful not only for engaging business users from the start, but also for surfacing limitations early on in big data initiatives and delivering some quick wins to demonstrate business benefits.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Hiring and retaining workers with big data skills"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Hiring and retaining workers with big data skills</h2> <p>Finding workers with the required skills is another common challenge -- and growing AI use adds new requirements for expertise in designing, training and supervising AI models. But data scientists and other analytics professionals with AI skills are in high demand, as are data engineers and workers skilled in <a href="https://www.techtarget.com/searchdatamanagement/feature/Building-a-strong-data-analytics-platform-architecture">deploying and managing data platforms</a>.</p> <p>In addition, technical skills alone aren't enough. Data teams also must be able to identify risks, manage internal expectations and resolve issues, said Pablo Listingart, founder and executive director of ComIT and Comunidad IT, charitable organizations that provide free IT training programs in Canada and Argentina. "Many big data initiatives fail because of incorrect expectations and faulty estimations that are carried forward from the beginning of the project to the end," he noted.</p> <p>Vojtech Kurka, co-founder and head of R&amp;D at customer data platform vendor Meiro, said creating the right culture helps attract and retain skilled workers. Kurka initially thought Meiro could solve its data problems with simple SQL and Python scripts. But he later realized that to meet its goals, the company needed to hire people with more advanced data skills and keep them satisfied and motivated.</p> <p>Organizations can also partner with providers of AI, analytics, data management and software development services to fill big data skills gaps. In some cases, that's faster and less expensive than hiring new employees. But data leaders should carefully evaluate a provider's costs and capabilities and assess whether internal hiring is a better long-term option.</p> </section> <section class="section main-article-chapter" data-menu-title="8. Keeping costs from getting out of control"> <h2 class="section-title"><i class="icon" data-icon="1"></i>8. Keeping costs from getting out of control</h2> <p>Another common challenge is avoiding what David Mariani, co-founder and CTO of semantic layer platform vendor AtScale, called the "cloud bill heart attack."</p> <p>Many enterprises use existing data consumption metrics to estimate the computing costs of new big data infrastructure, but expanded access to richer, more granular data sets often increases user demand for computing resources. Cloud systems that elastically scale to handle higher data processing and analysis workloads will <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Why-a-dashboard-audit-matters-before-BI-cleanup">drive up costs unexpectedly</a> if companies underestimate their resource needs.</p> <p>On-demand pricing models can also increase costs if the use of big data systems isn't managed effectively.&nbsp;Fixed-resource pricing alleviates that problem, but doesn't completely solve it: Poorly written applications that consume excessive resources block other workloads from running if the specified usage limit is reached. "I've seen several customers where users have written $10,000 queries due to poorly designed SQL," Mariani said. Data teams need to implement fine-grained query controls to prevent that.</p> <p>Rehl said data leaders should also raise the cost issue upfront with business and data engineering teams when planning big data deployments to ensure organizations budget appropriately for required computing resources and include effective cost controls.</p> </section> <section class="section main-article-chapter" data-menu-title="9. Governing big data environments"> <h2 class="section-title"><i class="icon" data-icon="1"></i>9. Governing big data environments</h2> <p>Without effective data governance, "much of the benefit of broader, deeper data access can be lost," Mariani said. But data governance issues become harder to address as big data applications expand across systems. Cloud architectures that make it more feasible for enterprises to collect and store ever-increasing volumes of raw, unaggregated data compound governance challenges.</p> <p>Lax data governance reduces the accuracy of analytics and AI outputs and allows protected information to creep into applications that shouldn't include it, creating compliance risks. In addition to the <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">data protection and privacy laws</a> that mandate strong governance, <a href="https://www.techtarget.com/searchenterpriseai/feature/AI-regulation-What-businesses-need-to-know">AI regulations</a> are becoming a factor, Farmer said. For example, under the EU AI Act, qualifying organizations deploying AI systems classified as high-risk must meet a set of data governance and management requirements starting in August 2026.</p> <p>Investing time upfront to identify and&nbsp;manage big data governance issues&nbsp;makes it easier to provide self-service data access without requiring direct oversight of each new use case. Treating data as a product with built-in governance rules also helps prevent usage and compliance issues.</p> </section> <section class="section main-article-chapter" data-menu-title="10. Ensuring that AI tools produce trustworthy results"> <h2 class="section-title"><i class="icon" data-icon="1"></i>10. Ensuring that AI tools produce trustworthy results</h2> <p>Generative AI (GenAI) and agentic AI tools <a href="https://www.techtarget.com/searchdatamanagement/feature/How-agentic-AI-governance-tackles-data-security-challenges">amplify data management and governance issues</a> in big data systems. For example, AI agents configured to autonomously monitor, analyze and act on data can create cascading errors and compliance problems without proper oversight.</p> <p>Comprehensive training and ongoing supervision are required to ensure that AI's actions are accurate, unbiased and trustworthy, said Michael O'Malley, senior vice president of strategy and growth at Customer Analytics LLC, an AI, analytics and data engineering services provider. "Agents and generative AI are powerful tools," O'Malley said. "But just owning an expensive hammer doesn't make you a master carpenter."</p> <p>Data quality is also a key consideration: An AI agent is only as reliable as the data it analyzes, Silipo noted. In addition, the models that underpin GenAI and agentic AI tools must be updated when new business trends or scenarios inevitably emerge. Otherwise, the tools won't be able to adapt, leading to flawed analytics and actions.</p> <p><strong>Editor's note:</strong> <em>This article was updated in May 2026 for timeliness and to add new information.</em></p> <p><em>George Lawton is a journalist based in London. Over the last 30 years, he has written more than 3,000 stories about computers, communications, knowledge management, business, health and other areas that interest him.</em></p> </section> Big data initiatives pose various challenges for data leaders and teams. Here are 10 common ones and advice on overcoming them to ensure deployments are successful. https://cdn.ttgtmedia.com/visuals/searchCloudProvider/service_management/cloudprovider_article_008.jpg https://www.techtarget.com/searchdatamanagement/tip/10-big-data-challenges-and-how-to-address-them Fri, 22 May 2026 22:40:00 GMT <p>As self-service analytics blooms and teams prioritize speed over consistency, dashboard sprawl can take root, sowing confusion among users and adding extra costs and work.</p> <p>What begins as convenience can quietly become complicated: Duplicate tools drive licensing and platform overhead, competing data definitions erode trust, and leaders struggle to show the benefits of their analytics investments.</p> <p>As organizations struggle with dashboard sprawl challenges, data teams are increasing spending, which can compound the problem. Mordor Intelligence predicts the global data analytics market will reach $108.79 billion in 2026, up from $82.33 billion in 2025. In many organizations, dashboards multiply faster than governance rules, ownership assignments and internal standards can <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">keep up to maintain consistent metrics </a>across the enterprise.</p> <section class="section main-article-chapter" data-menu-title="When dueling dashboards create doubt"> <h2 class="section-title"><i class="icon" data-icon="1"></i>When dueling dashboards create doubt</h2> <p>Matt Arellano has seen what happens when dashboards outpace governance.</p> <p>Arellano, a senior vice president at IT services and consulting company Genpact, worked with a global retailer whose U.S. and European sales teams used separate analytics systems with dashboards that calculated gross sales differently. As a result, the teams <a href="https://www.techtarget.com/searchdatamanagement/feature/Controlling-data-sprawl-requires-governance-discipline">reported inconsistent numbers</a> to the finance department, "leaving a numerical problem for finance to figure out," Arellano said.</p> <p>The conflicting calculations also meant the company didn't spend its marketing dollars efficiently and made it appear that the U.S. and European markets were performing differently, even though they were not, Arellano said.</p> <p>Organizations have invested heavily in data platforms and business intelligence (BI) technologies , betting that analytics would yield clearer insights and, ultimately, <a href="https://www.techtarget.com/searchbusinessanalytics/feature/Real-time-edge-analytics-use-cases-for-business">faster, more informed business decisions</a>. Yet trust in analytics results is slipping in many organizations, as is the value of data analysis, often due to dashboard sprawl.</p> <p>The problem is widespread, said Boris Evelson, a vice president and principal analyst at Forrester Research.</p> <p>"The majority of organizations are still in the broken dashboard phase," Evelson said. "And the elephant in the room here is that as a result, no one can figure out what their ROI is on their data and analytics investments. This is one of the top complaints we hear [from clients]. They don't know how much they've gained for all their work. They're saying, 'We've got all these dashboards, but how do we know they're bringing us business value?'"</p> </section> <section class="section main-article-chapter" data-menu-title="Why dashboard sprawl is rising"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Why dashboard sprawl is rising</h2> <p>The typical organization has numerous dashboards. Several factors have contributed to their growth, according to experts.</p> <p>First is the <a href="https://www.techtarget.com/searchbusinessanalytics/feature/The-future-of-business-intelligence-Top-trends">democratization of data</a>, which extended access to data to employees across organizations rather than limiting it to skilled analytics professionals, such as BI developers and data scientists.</p> <p>Second is the large enterprise investment in analytics tools that include dashboards.</p> <p>Third is the growing availability of technologies that enable self-service users to easily create their own dashboards. As a result, many dashboards are built outside formal governance channels, which can lead to shadow analytics. Some dashboards are built by a particular user or team, used once and forgotten, becoming so-called zombie dashboards that burn up computing resources and licensing budgets without any ROI.</p> <p>"The democratization of data and business intelligence, along with the low barrier of entry for creating new dashboards, has accelerated the sheer number of dashboards and reports that are produced and consumed," said Nick Kramer, principal of applied solutions at consultancy SSA &amp; Company.</p> <p>But it's not just the number of dashboards within an organization that's a problem, Kramer and others said. The bigger issue is inconsistent data definitions across dashboards -- a problem that stems from <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">governance gaps</a>.</p> <p>"Companies may have data governance, but they don't extend it into their BI layer," Kramer said. "And you have a ton of dashboards being built, and they're also relatively ungoverned. So, you have different dashboards with an overlap in what they report, and they do it differently."</p> </section> <section class="section main-article-chapter" data-menu-title="How dashboard sprawl hurts organizations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How dashboard sprawl hurts organizations</h2> <p>Dashboard sprawl negatively impacts an organization in multiple ways.</p> <p>To start, dashboard sprawl adds unnecessary costs, said Doug Leal, vice president of data and analytics consulting at CGI, an IT and business consultancy. Duplicate tools and zombie dashboards <a href="https://www.techtarget.com/searchcio/feature/Predictable-IT-spending-in-an-unpredictable-economy">incur excess licensing fees</a>, require IT support and consume compute resources without providing sufficient -- or sometimes any -- benefits to justify the money spent on them.</p> <p>Dashboard sprawl also has indirect costs, Leal explained. Because data definitions vary across multiple dashboards, different workers and teams produce conflicting reports. That leads to disputes over who has the right insights.</p> <p>"There's an inability to make a decision. That leads to a lack of agility, and you become stagnant as an organization because you can't move forward," Leal said.</p> <p>"The paralysis that comes with not knowing what the real answer is, is real," Kramer added.</p> <p>That, in turn, wears away employee and executive trust in the organization's data, even if the <a href="https://www.techtarget.com/searchdatamanagement/opinion/Most-data-governance-wasnt-built-for-AI">data quality</a> itself is sound.</p> <p>"When you have multiple dashboards reporting the same metric, but they're not reporting the same number, [people] then don't trust the data. And the cost to turn that around and get them to trust the data again is huge," Leal said.</p> </section> <section class="section main-article-chapter" data-menu-title="How leaders can rein in dashboard sprawl"> <h2 class="section-title"><i class="icon" data-icon="1"></i>How leaders can rein in dashboard sprawl</h2> <p>Dashboard sprawl is not an indictment against dashboards themselves, nor even having a high number of them in an organization.</p> <p>"Dashboards absolutely bring value. They are absolutely useful, and we cannot operate without them," Evelson said. "But there are too many dashboards created with expensive dollars that aren't being used or that are being used that provide signals without insight, so you can't act on them. That's the problem."</p> <p>Evelson stressed that an organization shouldn't eliminate dashboards just to reduce the number; instead, it should aim to better govern them.</p> <p>The goal of better governance is to reduce duplicate tools, <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Top-11-business-intelligence-challenges-and-how-to-overcome-them">enforce consistent data definitions</a> and data quality, and retire dashboards when they're no longer needed or no longer provide value.</p> <p>Those are challenging tasks, experts said.</p> <p>To execute them effectively, experts said organizations must have a strong data governance operating model that includes metadata management and a semantic <a target="_blank" href="https://www.youtube.com/watch?v=K_VPF9URZo4" rel="noopener">layer</a>, and that enforces data stewardship and ownership. They also need asset management to inventory dashboards and a searchable catalog so teams can reuse existing ones.</p> <p><em>Mary K. Pratt is an award-winning freelance journalist specializing in enterprise IT, cybersecurity strategy and data management.</em></p> </section> The proliferation of dashboards, coupled with conflicting data definitions, exposes governance issues in organizations and reduces the ROI of analytics investments. https://cdn.ttgtmedia.com/rms/onlineimages/collab_a120705009.jpg https://www.techtarget.com/searchbusinessanalytics/feature/How-dashboard-sprawl-challenges-upend-enterprise-analytics Fri, 22 May 2026 13:32:00 GMT <p>Data sovereignty has expanded from a regulatory compliance matter into a broader risk management challenge as IT environments and global pressures grow more complex.</p> <p>Addressing data sovereignty once meant complying with the laws governing data from its country of origin to others where it's processed or stored. But sovereignty concerns now also encompass AI models, supply chain disruptions and military conflicts. Chief data officers, CIOs and CISOs must consider the expanding scope of sovereignty and <a href="https://www.techtarget.com/searchenterpriseai/tip/How-to-navigate-data-sovereignty-for-AI-compliance">how to navigate a multitude of variables</a> subject to rapid, unpredictable change. Sovereignty strategies need to maintain control and flexibility across jurisdictions while also balancing cost, compliance and performance.</p> <p>Experts recommend aligning sovereignty goals with data sensitivity levels and treating sovereignty as a risk management endeavor. The results, ideally, will not only meet regulatory requirements but also strengthen organizational resilience and technology independence.</p> <section class="section main-article-chapter" data-menu-title="The new world of data sovereignty"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The new world of data sovereignty</h2> <p>The task for enterprise data leaders is challenging. On the traditional compliance side, they face numerous regulations that address elements of sovereignty but do not fully define it. Organizations navigating jurisdictional control of data do so without a single, authoritative global standard comparable to the EU's GDPR, which serves as a de facto data privacy standard.</p> <p>"A clear GDPR for sovereignty doesn't exist," said Dario Maisto, a senior analyst at Forrester. "There is no legislation whatsoever in the world that tells you what sovereignty is and isn't."</p> <p>Maisto said that some regulations, such as India's Digital Personal Data Protection Act, 2023, touch on sovereignty within specific domains but don't define it more broadly. The <a href="https://www.computerweekly.com/news/366549932/India-gets-ready-for-new-data-protection-regime">DPDPA</a> applies only to digital personal data or data digitized after being collected, he noted. It focuses on data protection and permits cross-border data flows, except to restricted regions, Maisto added.</p> <p>Another complication for data sovereignty efforts is physical risk to data centers and other technology infrastructure. <a href="https://www.techtarget.com/searchdatabackup/feature/Geopolitics-reshapes-data-protection-plans">Geopolitical conflict</a> and natural disasters can strip organizations of jurisdictional control even when data is stored and processed in accordance with local laws.</p> <p>For example, Iran on Feb. 28 began launching salvos of drone and missile attacks at the Gulf States in response to the U.S. and Israeli strikes against the country. The next day, AWS reported damage to data centers in Bahrain and the United Arab Emirates caused by Iranian <a target="_blank" href="https://health.aws.amazon.com/health/status" rel="noopener">drones</a>. The attacks severely disrupted services at the facilities, and the cloud provider urged customers to migrate workloads to other AWS Regions in the U.S., Europe or Asia-Pacific, "as appropriate for your latency and data residency requirements."</p> <p>"The geopolitics have shifted tremendously in the last few months," said Ron Babin, adjunct research advisor for IDC's IT Executive Programs. "As a CIO, that's something that you rarely have had to worry about -- having your data center as a target of military forces."</p> <p>The threat to IT infrastructure and data, <a href="https://www.computerweekly.com/feature/Breaking-the-stranglehold-Responses-to-data-sovereignty-risk">concerns about dependencies</a> on foreign tech providers, and ambiguous regulations have drastically changed the nature of sovereignty.</p> <p>"There is a paradigm shift going on in the market," Maisto said. "It used to be a data protection problem [and] a privacy-related issue. Now, instead, it is really about risk management."</p> </section> <section class="section main-article-chapter" data-menu-title="Developing a sovereignty strategy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Developing a sovereignty strategy</h2> <p>Against this backdrop, industry executives and analysts outlined frameworks and methods to help IT leaders operationalize sovereignty strategies.</p> <p>Kyndryl, an IT infrastructure firm, in April launched a suite of advisory, implementation and managed services focused on sovereignty. Logan Wolfe, partner for global enterprise transformation, AI and sovereign tech strategy at Kyndryl, said the services reflect the new directions of sovereignty.</p> <p>"The conversation has shifted, fundamentally so, from a regulatory compliance basis -- a box-ticking exercise, if you will -- to how can we apply [sovereignty principles and strategy] as a fundamental risk framework to operate resiliently," he said.</p> <p>The Kyndryl framework spans three dimensions: data sovereignty, operational sovereignty and technological sovereignty. The data side focuses on where data is stored and processed and <a href="https://www.techtarget.com/searchapparchitecture/tip/Privacy-compliance-and-governance-are-changing-development">who can access it</a>, Wolfe said. It also covers which regulatory regimes apply to a particular organization. Operational sovereignty involves an organization's ability to operate, maintain and recover systems without undue external dependencies. Technological sovereignty centers on limiting its dependence on technologies controlled or subject to interference by foreign governments.</p> <p>A readiness assessment evaluates a business's posture across the three dimensions. This should spur a strategic discussion and produce an action plan on how to achieve specific sovereignty goals. Wolfe said those goals have different drivers, such as data and AI, operational independence and regulatory compliance. Each goal will have a different action plan.</p> <p>IDC's framework focuses on sovereign AI, which includes data sovereignty. The framework <a target="_blank" href="https://www.idc.com/resource-center/blog/ai-sovereignty-risk-a-five-step-agenda-for-cios/" rel="noopener">covers</a> core technologies such as data models, infrastructure and AI chips and extends to include sourcing of model training data, supply chain relationships and regulatory compliance. Geopolitics is included as a constraint.</p> <p>Babin said CIOs, CDOs and other leaders need to determine where their organization stands regarding the jurisdictions where it operates and the applicable rules and regulations. That means understanding "the footprint of your organization, [including] customers, operating units and supply chains," he noted.</p> <p>In addition, the organization's industry -- whether financial services, transportation, power generation or others -- will have specific AI regulations to consider, Babin said.</p> <p>At Forrester, Maisto also cited <a href="https://www.techtarget.com/whatis/feature/Sovereign-AI-explained">AI's influence on data sovereignty approaches</a>. Data resides in an infrastructure, flows through a network, is used by SaaS offerings and feeds into AI workloads and large language models, he said. At that point, technology and business users operationalize the data. <br><br>"That's where data sovereignty actually develops into six different domains: data, infrastructure, network, software, AI and people," Maisto said.</p> <p>Across those domains, the specific sovereignty issue will drive a particular remedy, he added.</p> </section> <section class="section main-article-chapter" data-menu-title="Implementing sovereignty controls"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Implementing sovereignty controls</h2> <p>The strategy and planning work leads to the implementation stage, which focuses on controls -- the risk mitigation measures used to address sovereignty challenges.</p> <p>Cloud platform infrastructure, which houses data and workloads, is a critical area for establishing controls. The sovereign instinct might be to keep all data assets within a locally owned and hosted cloud.</p> <p>"A lot of people look at sovereignty and they think isolationism: 'We are just going to isolate as much as possible and create this digital fortress,'" Wolfe said.</p> <p>But instead of dropping public clouds entirely, an organization can pursue segmentation as a design pattern. A business in the EU, for instance, could identify and <a href="https://www.techtarget.com/searchsecurity/tip/Confidential-computing-use-cases-that-secure-data-in-use">segment sensitive data sets</a> and workloads restricted to EU jurisdiction with tighter operational controls, Wolfe said. Other data sets and workloads could continue to benefit from global cloud scale, he added.</p> <p>For example, a business might want to adopt or continue using a SaaS product that only runs on a particular hyperscaler's platform. Such cases make achieving full sovereignty a nightmare, Maisto said. Businesses operating in more than 100 countries, as some of his clients do, underscore the impossibility of the task, he added.</p> <p>Maisto's alternative is what he terms "minimum viable sovereignty," which relies on workload assessment. Here, data and IT leaders identify workloads that absolutely require <a href="https://www.forrester.com/blogs/digital-sovereignty-why-tech-execs-must-act-now/">sovereign controls</a>. The idea is to reserve the tightest control for the most sensitive workloads<i>, </i>while avoiding over-engineered protection for less sensitive ones.<b><i> </i></b>This method offers the ability to combine hyperscaler clouds, local cloud providers, private clouds and on-premises resources as appropriate.</p> <p>With minimum viable sovereignty, organizations must determine whether a sovereign cloud is an available option for their problems, Maisto noted. Sovereign capabilities from hyperscalers and other cloud providers are <a href="https://www.computerweekly.com/feature/Sovereign-cloud-and-AI-services-tipped-for-take-off-in-2026">entering the market</a> at a quickening pace, so leaders need to check regularly for new developments.</p> <p>If a sovereign cloud is available, the assessment shifts to cost. Maisto outlined some questions to consider: Is the sovereign offering more or less expensive than the non-sovereign version? If it's more expensive, does the sovereign option justify the cost?</p> <p>In addition, organizations should weigh cost trade-offs between hyperscalers and local providers. Achieving sovereignty with a hyperscaler might prove more costly than working with a local vendor, Maisto said. But even if the local vendor is cheaper, change management and migration costs would still need to be considered, he noted.</p> <p>Controls such as <a href="https://www.techtarget.com/searchdatabackup/tip/The-new-standards-for-cyber-resilient-backup-strategy">data vaulting and advanced encryption</a> also play a role in building a sovereign architecture. With data vaulting, isolated and protected copies are created for resilient recovery, Wolfe said. Advanced encryption coupled with customer-controlled key management provides sovereign control over key access, he added.</p> </section> <section class="section main-article-chapter" data-menu-title="Building AI models with sovereignty in mind"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Building AI models with sovereignty in mind</h2> <p>As AI becomes embedded in the bigger sovereignty picture, enterprises need to build AI models without moving raw data around by using techniques such as federated learning and secure multiparty computation. Fariba Wells, senior vice president of global government affairs and policy at Kyndryl, said she has seen some uptake for those methods.</p> <p>"Adoption is growing but remains selective -- concentrated in highly regulated sectors where data movement is tightly constrained," she said.</p> <p>Wells said the techniques align well with sovereignty objectives in that they allow AI models to be trained or refined without centralizing raw data -- directly reducing legal and regulatory exposure.</p> <p>"Federated learning is the more mature of the two in enterprise settings," Wells noted.</p> <p>That approach enables model training within local or sovereign environments while sharing only model updates or parameters across regions, she said. This lets organizations benefit from broader data patterns without violating data residency or access restrictions.</p> <p>Secure multiparty computation and related privacy-enhancing technologies "show real promise" but are currently being applied only in narrow use cases due to complexity, performance overhead and integration challenges, Wells said.</p> </section> <section class="section main-article-chapter" data-menu-title="Consistency and flexibility in sovereignty"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Consistency and flexibility in sovereignty</h2> <p>Sovereignty programs aim to meet local requirements. But can they also achieve some level of consistency regarding policies, metadata, access control and quality checks -- even when data must remain in separate regions?</p> <p>"The effective pattern is centralized governance with local enforcement," Wells said.</p> <p>That is, organizations <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">standardize governance policies,</a> metadata definitions and control frameworks globally, then enforce them locally within each jurisdiction to respect sovereignty constraints, she noted.</p> <p>Wells described centralized metadata and governance as the enabling layer. Here, business glossaries, lineage management and defined data stewardship roles give teams visibility into where data resides, how it's used and which policies apply, regardless of location, she said. <br><br>"The goal is not to eliminate fragmentation -- it's to orchestrate it," Wells said.</p> <p>As a result, organizations maintain control while preserving the operational flexibility needed to run analytics and <a href="https://www.techtarget.com/searchcio/feature/How-CIOs-can-beat-AI-challenges-A-top-researchers-view">AI at scale</a>, she said.</p> <p>Given the changing regulatory landscape and evolving technologies, adaptability is a requirement for sovereignty.</p> <p>"It's the optionality that is really key here," Wolfe said. "Every cloud and AI decision already has sovereignty implications. The advantage goes to leaders who make those decisions deliberately, instead of making decisions by inaction."<br><br><i>John Moore is a freelance writer who has covered business and technology topics for 40 years. He focuses on enterprise IT strategy, AI adoption, data management and partner ecosystems.</i></p> </section> Geopolitical conflict and outages can upend assumptions about where data is controlled and accessed, pushing leaders to plan for jurisdictional risk, resilience and recovery. https://cdn.ttgtmedia.com/rms/onlineimages/strategy_a296343799.jpg https://www.techtarget.com/searchdatamanagement/feature/Data-sovereignty-expands-beyond-compliance-boundaries Fri, 15 May 2026 14:02:00 GMT <p>Power, not compute capacity, is becoming the primary constraint on where AI workloads and supporting platforms run -- and when they can go live.</p> <p>For enterprise data and analytics leaders, power concerns are no longer confined to data center facility planning. They are now a data architecture issue affecting AI deployments, platform scaling and resilience plans.</p> <p>Global electricity demand from data centers is projected to more than double by 2030, according to the International Energy Agency. Gartner forecasts AI workloads will use 44% of data center power by then. While the demand for power is surging due to AI, grid interconnection delays in key markets limit the availability of new data center capacity.</p> <p>Together, those forces are creating a new reality of power-constrained data architecture, where time-to-power becomes a core factor in AI infrastructure decisions alongside cost, latency and performance. The question is no longer just where infrastructure is cheapest or fastest, but <a href="https://www.techtarget.com/searchenterpriseai/feature/Is-GenAI-villain-and-hero-in-data-center-power-drama">where AI workloads can be powered</a>, scaled and protected from disruptions.</p> <p>Interconnection queues, equipment lead times and constrained regional capacity mean that power access is no longer a given but a planning variable. Organizations that do not account for it early in site selection, workload design and procurement face project delays, cost overruns and operational outages they could have anticipated. The impact is most visible in infrastructure planning, where power delivery timelines are beginning to shape data architecture decisions.</p> <section class="section main-article-chapter" data-menu-title="Time-to-power reshapes architecture and procurement"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Time-to-power reshapes architecture and procurement</h2> <p>The timeline from site selection to an energized data center facility has pushed well beyond typical project assumptions, and the delays are coming from multiple directions at once. An interconnection agreement does not necessarily mean power will arrive soon, even if data platforms are ready to deploy.</p> <p>Dell'Oro Group's February 2026 forecast projects the worldwide data center physical infrastructure market to grow by a mid-teens percentage each year through 2030, surpassing $80 billion by 2030. However, the market research firm said power scarcity is driving increased demand for on-site generation as data center operators work around grid constraints. On-site approaches are shifting from a contingency to a practical necessity for large facilities, it noted.</p> <p>Interconnection delays for power plants are also affecting data center timelines. A December 2025 report from Lawrence Berkeley National Laboratory found that power generation and storage capacity projects built in the U.S. from 2018 to 2024 took a median of more than four years to go from the interconnection request to commercial operation. In Northern Virginia, the country's largest data center market, transmission expansion is now a multiyear planning issue, extending to eight years or more. Timelines are comparable in parts of Europe, including the U.K. and Germany.</p> <p>The <a href="https://www.techtarget.com/searchenterpriseai/feature/Communities-call-for-transparency-in-AI-data-center-deals">pressure is not limited to hyperscalers</a> and large colocation providers that are securing hundreds of megawatts for AI deployments. Alex Cordovil, research director at Dell'Oro Group, said enterprise teams are already seeing downstream effects.</p> <p>"When hyperscalers and colocators absorb the lion's share of available utility capacity in a given market, enterprise teams face a knock-on effect, whether that's longer lead times for their own on-premises expansions, reduced availability at their preferred colocation facilities, or less favorable contract terms," Cordovil said.</p> <p>Energization dates -- when a facility can draw grid power -- now carry as much weight in site selection as network latency or real estate cost. Bloom Energy's 2025 data center power survey found that 84% of respondents ranked power access among their top three site-selection considerations. When timelines slip, organizations might need to move AI workloads to other regions, delay deployments or accept limited power access with potential interruptions during peak demand.</p> </section> <section class="section main-article-chapter" data-menu-title="The transition to on-site power and grid-interactive operations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>The transition to on-site power and grid-interactive operations</h2> <p>In addition to on-site power infrastructure they control directly, data center operators are turning to grid-interactive operations that adjust electricity use based on overall demand, reflecting a shift toward power-aware data platform design.</p> <p>Gartner expects energy alternatives, such as green hydrogen and <a href="https://www.techtarget.com/sustainability/feature/EmTech-MIT-2025-looks-at-the-future-of-US-nuclear-energy">small modular nuclear reactors</a>, to become more viable for data center microgrids by the end of the decade. Dell'Oro Group's quarterly tracking shows that worldwide data center physical infrastructure spending increased 20% year-over-year in the fourth quarter of 2025. Power distribution was one of the stronger growth areas, driven by demand for higher-density AI infrastructure.</p> <p>The shift is playing out across five technology categories:</p> <ul class="default-list"> <li><b>Strategic siting.</b> Colocating capacity near existing grid-connected infrastructure can help shorten deployment timelines.</li> <li><b>Microgrids.</b> Combining on-site generation with battery storage, microgrids enable facilities to operate independently of the grid during stress events to reduce the exposure of critical data workloads to instability.</li> <li><b>Fuel cells</b>. Solid-oxide fuel cells offer a faster path to primary power than traditional grid connections, deploying in weeks rather than years. They also run at significantly higher electrical efficiency than diesel generators, but buyers should assess cost, emissions reporting and runtime assumptions.</li> <li><b>Battery energy storage systems. </b>BESS can come online during outages, reducing dependence on diesel generators and supporting grid independence when a utility's power supply is interrupted.</li> <li><b>Workload flexibility.</b> AI and data workloads are increasingly designed to shed or shift load during peak grid stress without taking systems offline. Training workloads that tolerate latency are especially suited to this model.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Provider questions for power-constrained AI workloads"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Provider questions for power-constrained AI workloads</h2> <p>Understanding what data center operators are deploying is only part of the <a target="_blank" href="https://www.belfercenter.org/research-analysis/ai-data-centers-us-electric-grid" rel="noopener">equation</a>. Data leaders and other enterprise buyers also need to know whether the cloud and data platform providers they are evaluating are keeping pace.</p> <p>Cordovil said most buyers are still evaluating providers on the wrong metrics. Power usage effectiveness, redundancy tier and price per kilowatt are baseline measures. The questions that matter now, he added, are about power pathway resilience, future-proofing and flexibility:</p> <ul class="default-list"> <li>What is the provider's contracted utility capacity versus what is currently energized?</li> <li>How much headroom exists for <a href="https://www.techtarget.com/searchdatacenter/tip/Data-center-trends-to-watch">density growth</a>, not just at the facility level but at the rack and row level?</li> <li>What is the provider's strategy for on-site generation, grid interactivity and demand response participation?</li> <li>How flexible is the infrastructure to accommodate shifting power and cooling profiles as AI architectures evolve? Can it scale incrementally, or does it require a full replacement?</li> <li>What does the power roadmap look like over the next five years, not just what is available today?</li> </ul> <p>Tony Harvey, an analyst at Gartner, said teams should also ask whether promised power delivery is firm or can be reduced during grid stress events. If it's the latter, they should ask what conditions would trigger a reduction, how often it could happen and whether critical workloads can be protected from curtailment. If on-site generation is the backup plan, buyers should confirm whether it can carry the full power load and for how long.</p> </section> <section class="section main-article-chapter" data-menu-title="Planning ahead for power grid risk"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Planning ahead for power grid risk</h2> <p>Even with better provider selection, power risk remains a resilience issue for enterprise data teams. Power remains the leading cause of impactful data center incidents, according to Uptime Institute's annual outage analysis report for 2025. On-site generation and workload flexibility reduce exposure but do not eliminate risk.</p> <p>A complete resilience posture requires action across three areas:</p> <ul class="default-list"> <li><b>Physical separation across regions is the foundation.</b> Logical separation within a single facility provides no protection against a shared power failure. The baseline is cross-region replication of applications, data and logs across sites on separate feeds from separate providers, with failover tested under load rather than on paper.</li> <li><b>Generator and UPS testing must reflect actual failure scenarios.</b> A quarterly exercise that never reaches full load does not validate performance. Best practices include full load-bank testing quarterly; an annual black-start test where generators start cold and an uninterruptible power supply (UPS) bridges the gap; and verification that security systems and access controls hold during switchovers. Uptime Institute found failure to follow established procedures is the leading addressable cause of outages, making process review more impactful than new hardware.</li> <li><b>Workload classification is a prerequisite for everything else.</b> Teams that have mapped AI workloads by delay tolerance can negotiate demand-response agreements with utilities. That classification also provides a plan for shifting workloads across regions and power providers when a primary site faces constraints.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Terms should reflect power availability risk"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Terms should reflect power availability risk</h2> <p>AI workloads are increasingly moving closer to enterprise infrastructure for model fine-tuning, inference at the edge and retrieval-augmented generation, and the data architectures underpinning them are evolving quickly. The power and cooling envelope an organization needs to support platforms and applications today might look very different in 18 months.</p> <p>Power risk spans site selection, procurement and operations, but the contract with a provider defines an organization's exposure. Harvey said many buyers don't realize the terms are negotiable.</p> <p>He recommended that they:</p> <ul class="default-list"> <li>Require proof of a signed interconnection agreement, completed required studies and a confirmed energization date.</li> <li>Define delivery requirements at the rack power distribution unit level, including redundant feeds and completed full-load testing.</li> <li>Make the colocation provider responsible for utility delays where possible.</li> <li><a href="https://www.techtarget.com/searchsecurity/tip/What-CISOs-should-look-for-in-cyber-insurance-policy-fine-print">Review the force majeure clause</a> and explicitly exclude utility delays from its scope.</li> <li>Attach penalties to non-delivery tied to documented business impact.</li> <li>Require zero curtailment for critical loads backed by a service-level agreement with financial penalties.</li> </ul> <p>Getting the terms right is necessary, but it is only the starting point for data leaders. The broader risk of treating power as an afterthought is strategic, not just operational. <br><br>"Buyers who treat power as a static line item on a colocation contract are going to find themselves either capacity-constrained or locked into infrastructure that can't adapt as AI workloads scale and shift," Cordovil said.</p> <p><em>Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.</em></p> </section> Rising power demands and grid interconnection delays are hampering enterprise AI efforts and altering data strategies, workload placement and resilience planning. https://cdn.ttgtmedia.com/rms/onlineimages/location_g580502719.jpg https://www.techtarget.com/searchdatamanagement/feature/Power-constrained-data-architecture-curbing-AI-ambitions Wed, 13 May 2026 12:12:00 GMT <p>Vector databases have moved from a niche component of generative AI infrastructure to standard tooling across data-intensive industries. The technology now supports a range of enterprise applications across industries, with use cases primarily found where similarity-based retrieval outperforms traditional database queries.</p> <p><a href="https://www.techtarget.com/searchdatamanagement/tip/Vector-vs-graph-vs-relational-database-Which-to-choose">Unlike traditional relational databases</a> that store data in tables with rows and columns, vector databases represent unstructured data -- text, images, audio -- as points in a multidimensional space, then retrieve results by similarity rather than exact matching. Algorithms such as approximate nearest neighbor make this practical at scale, which is why the approach has become foundational to generative AI applications.</p> <p>The following 10 use cases for vector databases illustrate how vector databases are delivering value across industries today.</p> <section class="section main-article-chapter" data-menu-title="1. Natural language processing"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Natural language processing</h2> <p>Vector databases capture complex semantic relationships that traditional data models miss, <a href="https://www.techtarget.com/searchdatamanagement/opinion/Why-data-semantics-matters-for-context-aware-systems">making them well-suited</a> to language tasks such as sentiment analysis and translation. Embedding-based representations encode context and tone in ways dictionary lookups can't, supporting more accurate classification, search and machine translation.</p> <p>The meaning of words depends greatly on context and cultural nuances. Embeddings recognize that "buy," "purchase" and "acquire" share meaning while differing slightly in context. Phrases such as "splurged on" carry sentiment beyond their literal definitions that lexical matching misses entirely.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Customer support"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Customer support</h2> <p>People who have experienced long waits with a pressing issue on a customer service call know that sentiment analysis is a critical skill for human customer service representatives. For online customer service, chatbots now feel a <a href="https://www.techtarget.com/searchcustomerexperience/feature/In-2023-generative-AI-made-inroads-in-customer-service">little closer to human interaction</a>. A knowledge base using vectors can analyze and respond to customer support tickets by accurately categorizing issues. The bot can also note underlying sentiment using techniques similar to natural language processing. Responses and prioritization are handled more satisfactorily for customers.</p> <blockquote class="main-article-pullquote"> <div class="main-article-pullquote-inner"> <figure> [Vector databases] are reshaping how people interact with and benefit from the vast amounts of data generated in the contemporary world. </figure> <i class="icon" data-icon="z"></i> </div> </blockquote> </section> <section class="section main-article-chapter" data-menu-title="3. Image and video recognition"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Image and video recognition</h2> <p>Vector databases convert the pixel data of images and videos into vector representations. Vector representations of images and video enable similarity-based search across visual content such as facial recognition, object detection and scene understanding with high accuracy.</p> <p>For example, an e-commerce site can tag products in images to enable shoppers to perform visual searches. A social media platform can also use the technique for content moderation, detecting policy violations in visual content to quickly flag explicit content in videos and images at scale without needing human case-by-case review.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Financial services fraud detection"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Financial services fraud detection</h2> <p>Representing transactions as vectors across many dimensions -- time, location, amount and the relationship between transactions -- makes it possible to detect subtle, nonlinear patterns typical of fraudulent behavior. Banks can identify anomalous vectors with more nuance than traditional rules-based fraud models, catching novel patterns that fixed rules miss. These new capabilities result in fewer false declines on legitimate transactions and stronger fraud detection for banks and card issuers.</p> </section> <section class="section main-article-chapter" data-menu-title="5. E-commerce product recommendations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. E-commerce product recommendations</h2> <p>E-commerce sites already deliver personalized product recommendations by analyzing customer browsing and purchase history. Data mining can identify correlations, but is too slow to respond to customer behavior on the website in near real time. Vector representations let recommendation systems match a shopper's current activity with similar products, customers and past sessions much faster, enabling e-commerce sites to offer a more engaging and insightful experience.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Autonomous vehicles"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Autonomous vehicles</h2> <p>Autonomous vehicle developers use vector databases in development workflows rather than the driving loop itself. Sensor logs from cameras, lidar, and radar captured across millions of fleet miles are encoded as vectors and stored for similarity search, allowing analysts to retrieve scenes resembling specific patterns <a href="https://www.techtarget.com/sustainability/feature/Breaking-the-cycle-of-algorithmic-bias-in-AI-systems">such as pedestrians</a>, traffic signals and obstacles.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Medical diagnostics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Medical diagnostics</h2> <p>The same approach applies to medical imaging. Scans such as MRIs and X-rays are encoded as vectors and compared against large data sets of cases with known conditions. The system can reveal patterns imperceptible to humans and help make an accurate diagnosis. As a result, doctors could accurately identify early indicators of cancer or other serious illnesses without invasive tests.</p> </section> <section class="section main-article-chapter" data-menu-title="8. Biometrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>8. Biometrics</h2> <p>Anyone who has recently traveled through an airport, applied for a driver's license or opened a bank account in person is familiar with biometric security. Vector analysis excels at biometric pattern recognition from fingerprints, facial characteristics and other identity attributes captured digitally for authentication. It converts slight but distinctive qualities to vectors. The result is a system that is much less likely to be confused by cuts on a person's fingers, haircuts, makeup or natural aging.</p> </section> <section class="section main-article-chapter" data-menu-title="9. Media recommendations"> <h2 class="section-title"><i class="icon" data-icon="1"></i>9. Media recommendations</h2> <p>Entertainment services, such as movie or music <a target="_blank" href="https://www.moltencloud.com/blog/film-industry/how-streaming-platforms-are-using-artificial-intelligence-in-2024" rel="noopener">streaming platforms</a>, make recommendations much like e-commerce sites, but with some important differences. Entertainment users often consume media in longer sessions -- such as watching a movie or binge-watching a series -- than online shoppers do. Instead of simple variables such as price and shipping options, recommendation systems use temporal dynamics such as new releases, viral content and even the mood of the viewer or listener. The process may be similar in principle, but in practice, vector databases are now essential for the complexity of media recommendations.</p> </section> <section class="section main-article-chapter" data-menu-title="10. Video games"> <h2 class="section-title"><i class="icon" data-icon="1"></i>10. Video games</h2> <p>In game development, vector databases support work that happens outside the real-time rendering loop. Studios use them to search large asset libraries by similarity, analyze player behavior telemetry for matchmaking and game balance, and, more recently, experiment with <a href="https://arxiv.org/html/2504.13928v1">LLM-driven character dialogue</a> that retrieves contextually relevant responses to player input.</p> <p>Vector databases are not just a technological innovation; they are reshaping how people interact with and benefit from the vast amounts of data generated in the contemporary world. Business applications are only scratching the surface of these models' potential. The flexibility and scale of vector databases offer intriguing possibilities for adaptive, engaging and analytically powered experiences in the future.</p> <p><b>Editor's note:</b> <i>This article was originally published in April 2024 and updated in May 2026 to reflect changes in how vector databases are used in production. </i></p> <p><em>Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> Vector databases have become the standard infrastructure for enterprise AI work. These 10 use cases for vector databases show where the technology delivers practical value. https://cdn.ttgtmedia.com/rms/onlineimages/iot_g1156740131.jpg https://www.techtarget.com/searchdatamanagement/tip/Top-industry-use-cases-for-vector-databases Mon, 11 May 2026 14:05:00 GMT <p>No data governance strategy is complete without a plan for tracking relevant KPIs and other metrics.</p> <p>Data governance metrics enable organizations to measure the effectiveness of governance processes across areas such as data quality, security, availability and usage. Collecting and analyzing metrics documents the <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">value of a data governance program</a>, helping data leaders and governance managers build an ongoing business case for it. If the program doesn't achieve desired outcomes, metrics highlight issues that must be addressed.</p> <p>Tracking governance metrics is also crucial for regulatory compliance. Although <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">data protection and privacy regulations</a> typically don't include explicit rules on collecting governance metrics, many require organizations to demonstrate that they take reasonable measures to maintain data security, privacy and, in some cases, quality. Metrics in these areas help show regulators and compliance auditors that a company is acting responsibly.</p> <p>Organizations track different sets of metrics based on their governance goals and requirements. However, they commonly incorporate these six types of metrics to get a comprehensive view of their data governance program.</p> <section class="section main-article-chapter" data-menu-title="1. Operational metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Operational metrics</h2> <p>The operational component of data governance centers on how an organization <a href="https://www.techtarget.com/searchdatamanagement/tip/6-key-steps-to-develop-a-data-governance-strategy">designs and implements its governance strategy</a>. The following are examples of metrics that provide insights into data governance operations:</p> <ul class="default-list"> <li><b>Total number of policies.</b> Tracking the number of data governance policies an organization has in place helps data leaders and business stakeholders understand the scope of governance operations and <a href="https://www.techtarget.com/searchenterpriseai/tip/How-AI-governance-and-data-privacy-go-hand-in-hand">ensure that required resources are available</a> to manage them.</li> <li><b>Governance assessment frequency and results.</b> Organizations should assess governance policies and procedures at fixed intervals, ideally once or twice per year. More frequent assessments might be required if data governance needs change quickly or if an organization undergoes a technological transformation -- such as increased AI adoption -- that has <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">major ramifications for governance policies</a>. Governance problems identified by other metrics can also trigger out-of-cycle assessments.</li> <li><b>Total data assets and data volumes.</b> Documenting the number of distinct data assets an organization has and the size of data volumes in different systems establishes resource baselines for data governance initiatives. Tracking changes in data assets and volumes also enables organizations to quickly scale governance operations when necessary.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="2. Data quality metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Data quality metrics</h2> <p>Low-quality data makes it hard to manage business operations effectively and generate accurate insights in analytics and AI applications to enable better decision-making. Data quality metrics <a href="https://www.techtarget.com/searchdatamanagement/tip/Evaluating-data-quality-requires-clear-and-measurable-KPIs">validate data reliability</a> for planned uses and highlight issues for data management and governance teams to address.</p> <p>Different metrics <a href="https://www.techtarget.com/searchdatamanagement/tip/6-dimensions-of-data-quality-boost-data-performance">measure data quality attributes</a> such as accuracy, completeness, consistency, timeliness and validity. The following are some commonly used ones:</p> <ul class="default-list"> <li><b>Number of duplicate data entries.</b> Duplicate entries in data sets lead to misleading analytics results and cause problems in business operations such as marketing, customer service and logistics.</li> <li><b>Data error rate.</b> Data errors such as corrupted data, typos and transposed numbers are another source of analytics and operational problems. Acceptable error rates often vary for different applications.</li> <li><b>Percentage of data records with missing or incomplete values.</b> Data that lacks required information due to missing values or incomplete fields also affects applications.</li> <li><b>Transformation error rate.</b> Tracking how often data transformations fail helps organizations identify resulting data quality issues, such as corrupted, inaccurate or incomplete data sets, as well as opportunities to improve <a href="https://www.techtarget.com/searchdatamanagement/tip/The-difference-between-data-cleansing-and-data-transformation">data transformation processes</a>.</li> <li><b>Number of data issues identified and corrected.</b> Tracking the number of data issues found and fixed within a specified period helps document the progress of <a href="https://www.techtarget.com/searchdatamanagement/feature/Proactive-practices-for-data-quality-improvement">data quality improvement efforts</a>.</li> </ul> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f.png"> <img data-src="https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f_mobile.png 960w,https://www.techtarget.com/rms/onlineImages/data_management-common_data_quality_metrics-f.png 1280w" alt="Graphic that lists commonly used data quality metrics." height="288" width="559"> <figcaption> <i class="icon pictures" data-icon="z"></i>Data management and governance teams use these metrics to assess data quality. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="3. Data availability and usage metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Data availability and usage metrics</h2> <p>Even high-quality data is only useful if business users and analytics teams can find and access it. Data availability and usage metrics provide insight into how effectively an organization uses data. Tracking them helps data management and governance teams identify technical issues in data platforms and problems with data discovery and accessibility.</p> <p>Examples of availability and usage metrics include the following:</p> <ul class="default-list"> <li><b>System uptime percentage.</b> Monitoring the uptime percentage of data platforms and analytics systems shows how often issues such as technical disruptions and cyberattacks interrupt data availability. This metric helps teams evaluate whether system changes or upgrades are required to prevent unexpected downtime.</li> <li><b>Average data latency.</b> Data latency measures how long it takes <a href="https://www.techtarget.com/searchbusinessanalytics/news/365534255/Data-pipelines-deliver-the-fuel-for-data-science-analytics">processing workflows and pipelines</a> to make data available for use after it's generated or updated. Tracking average latency shows whether initial availability goals -- often measured in seconds -- are being met.</li> <li><b>Data access frequency.</b> Measuring how often data is accessed helps determine whether its usage meets expected levels. Underutilization could indicate that data is difficult to access or that users aren't aware of available data resources due to inadequate training or <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">shortcomings in a data catalog</a>. It could also be a sign that a data set isn't relevant or useful to many people.</li> <li><b>Data update frequency.</b> Monitoring the number of updates and modifications to data also provides insights into its usage. Data writes represent a more active form of engagement than reads. Consequently, data sets with a high update frequency might require closer oversight from data quality analysts and governance teams.</li> <li><b>Number of unique users.</b> Tracking the number of distinct users accessing data also helps identify underutilized data assets and guide data management and governance activities.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="4. Data security and privacy metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Data security and privacy metrics</h2> <p>Effective data governance protects sensitive information from exposure or misuse that could create regulatory compliance issues and cause reputational damage. Data security and privacy metrics provide insights into the effectiveness of data governance policies and procedures and enable governance teams to proactively <a href="https://www.techtarget.com/searchdatamanagement/feature/Top-3-data-privacy-challenges-and-how-to-address-them">identify and address risks</a>.</p> <p>The following are some helpful data security and privacy metrics:</p> <ul class="default-list"> <li><b>Number of access violations.</b> Tracking how frequently unauthorized users attempt to access data helps teams evaluate the effectiveness of access controls and <a href="https://www.techtarget.com/searchsecurity/feature/How-to-create-a-data-security-policy-with-template">data security policies</a>. Unauthorized users include external attackers, employees with no legitimate need to access specific data sets or contractors who shouldn't have access to a company's internal data.</li> <li><b>Number of data breaches.</b> This is a separate metric for tracking incidents where unauthorized users <a href="https://www.techtarget.com/searchsecurity/tip/How-to-prevent-a-data-breach-10-best-practices-and-tactics">successfully access a company's data</a>.</li> <li><b>Number of privacy incidents.</b> This metric tracks improper access, use and handling of personal data that violates privacy laws and internal policies. In addition to data breaches involving personal information, privacy incidents include actions such as sharing data with unauthorized individuals and using it inappropriately in analytics and AI applications.</li> <li><b>Policy compliance rate.</b> This measures the percentage of data assets and processes that comply with data security and privacy policies, including data access controls, encryption standards, retention schedules and regulatory requirements.</li> <li><b>Number of policy exceptions.</b> Governance teams often grant temporary exceptions to data security and privacy policies. Tracking the number of exceptions helps manage them and identify areas where policies need to be updated. Teams can also track longer-term policy exemptions and compare the two numbers with the policy compliance rate to gain broader insights into data that isn't fully protected due to inadequate or unenforced policies.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="5. Data stewardship metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Data stewardship metrics</h2> <p>Data stewardship metrics focus on how effectively an organization engages with data governance practices and maintains governance standards. They track the <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-steward-responsibilities-fill-data-quality-role">activities of data stewards</a> who are directly responsible for managing data sets and overseeing the implementation of governance policies.</p> <p>Common data stewardship metrics include the following:</p> <ul class="default-list"> <li><b>Number of data stewards.</b> Tracking the number of data stewards assigned to data assets and comparing it to the number of assets and data volumes helps governance teams assess whether their organization has enough stewards -- or too many.</li> <li><b>Number of issues resolved.</b> The number of data governance problems that data stewards resolve is routinely tracked to monitor their activities and assess their productivity.</li> <li><b>Issue resolution time.</b> Similarly, tracking resolution times for data issues provides useful insights into data stewardship work and performance.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="6. Data literacy metrics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Data literacy metrics</h2> <p>Employees who understand the value of data and how to use it effectively are more likely to recognize the <a href="https://www.techtarget.com/searchdatamanagement/tip/Successful-data-operations-follow-a-data-governance-roadmap">need for strong data governance</a>. Organizations use data literacy metrics to assess literacy levels and how actively users participate in educational processes designed to <a href="https://www.techtarget.com/searchbusinessanalytics/feature/How-business-leaders-can-make-a-data-literate-culture-stick">create a more data-literate culture</a> that helps sustain governance initiatives.</p> <p>The following are examples of literacy metrics:</p> <ul class="default-list"> <li><b>Training engagement and completion rates.</b> Metrics that track user participation in <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">data literacy training programs</a> and broader data governance education -- such as the percentages of employees who enroll in and successfully complete training programs -- offer insight into engagement with governance initiatives. The percentage involved in governance activities is another useful metric.</li> <li><b>Data literacy scores.</b> The average scores from data literacy assessments conducted during training programs measure how well employees understand the material presented and their skills in areas such as data discovery, analysis and visualization. Tracking them helps governance teams assess both data literacy levels and the effectiveness of the training.</li> <li><b>Internal survey results.</b> User surveys conducted by governance teams provide both quantitative and qualitative metrics on data literacy and awareness in organizations.</li> </ul> <p><em>Chris Tozzi is a freelance writer, research adviser, and professor of IT and society who has previously worked as a journalist and Linux systems administrator.</em></p> </section> Implementing a data governance program isn't enough. Data leaders also need to track and analyze various metrics to evaluate its effectiveness and address shortcomings. https://cdn.ttgtmedia.com/visuals/searchCloudComputing/development/cloudcomputing_article_008.jpg https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more Fri, 08 May 2026 14:05:00 GMT <p>Low-code and no-code tools and AI-powered vibe coding accelerate data-related development work, but unexpected fallout can put enterprise analytics and AI applications at risk.</p> <p>These development approaches help shorten the distance from idea to impact by expanding who can innovate with data. In addition to IT teams, application owners and other citizen developers <a href="https://www.techtarget.com/searchenterpriseai/tip/Citizen-developers-are-redefining-enterprise-AI-development">can take the initiative</a> and address unmet data needs themselves. Wishful thinking by business users becomes creative problem-solving. Work gets done faster. But the speed of technology-enabled citizen development is colliding with data pipeline reliability issues in the enterprise. The downside of <a href="https://www.techtarget.com/searchsecurity/tip/Vibe-coding-security-risks-and-how-to-mitigate-them">unfettered development</a> is that hidden integration layers and ad hoc data pipelines create visibility gaps for data management teams. Despite the best intentions, the results can prove costly.</p> <p>A <a href="https://www.fivetran.com/blog/the-enterprise-data-infrastructure-benchmark-report-2026">survey</a> of 500 senior data and technology leaders conducted in late 2025 by data management vendor Fivetran found that legacy and DIY pipelines using traditional ETL processes break 30% to 47% more often than fully managed <a href="https://www.techtarget.com/searchdatamanagement/definition/Extract-Load-Transform-ELT">ELT</a> systems, resulting in an average monthly downtime of 60 hours. It also found that data engineers spend 53% of their time on pipeline maintenance, taking time away from supporting new analytics and AI uses.</p> <p>Anjan Kundavaram, chief product officer at Fivetran, said low-code, no-code and vibe coding all contribute to the DIY nature of many data conduits <a href="https://www.techtarget.com/searchenterpriseai/feature/Democratization-of-AI-creates-benefits-and-challenges">that escape oversight.</a></p> <p>"There's a hidden data integration layer, and there are hidden data pipelines," he said.</p> <p>Lina Vaskelė, chief risk and security officer at aircraft maintenance and repair services provider FL Technics, underscored this risk as businesses increasingly adopt low-code, no-code and AI-powered development tools.</p> <p>"I think the main issue is losing some visibility about the data, and how the outputs are used later,"&nbsp;she said.</p> <section class="section main-article-chapter" data-menu-title="Pain points in the pipeline"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Pain points in the pipeline</h2> <p>Attention to edge cases ranks among the key considerations in making data pipelines reliable, Kundavaram said. Potential wrinkles include changes to data sources.</p> <p>"There's a lot of incidental complexity in the actual data source, like Salesforce, Workday or any [application]," he said. "Those applications are changing over time. They are not static."</p> <p>An unmanaged, DIY pipeline might not be prepared to handle such changes. For instance, if a field in a data source's API is renamed, a pipeline built to look for that field will stumble when it encounters a different name. As a result, the pipeline delivers incomplete data to the target system, which could be a <a href="https://www.techtarget.com/searchdatamanagement/tip/Assemble-the-layers-of-big-data-stack-architecture">dashboard, data lake or data warehouse</a> -- or, increasingly, a data lakehouse that combines elements of the latter two platforms.</p> <p>Changes on the target side or in data throughput pose similar risks. A pipeline designed to handle a predictable capacity will break when there's a sudden spike. A source or target change combined with a spike in data volume could introduce cascading failures.</p> <p>No-code and vibe-coded pipelines often fail to anticipate those variables, Kundavaram said.</p> <p>"Some core parts of the data infrastructure are going to be very brittle. It's going to create a lot more outages," he said.</p> <p>The <a href="https://www.techtarget.com/searchhrsoftware/feature/Enterprise-AI-enters-its-operational-phase">rise of AI systems</a> only increases the risk of ungoverned data workflows. Hastily built and potentially hidden pipelines can feed flawed data into AI models, reducing their accuracy and eroding decision quality and trust.</p> <p>"The focus is shifting from just model performance to the reliability and transparency of the entire data ecosystem," said&nbsp;Amir Kazmi, chief technology and growth officer at Ralliant, which makes test and measurement tools, industrial sensors and other precision instruments. "The issue isn't dramatic model corruption as much as it is gradual decision degradation, where small inconsistencies compound over time."</p> <p>Kazmi said pipelines that are fragmented or not fully observable make it harder to trace how outputs are generated, validate the inputs and detect model drift.</p> </section> <section class="section main-article-chapter" data-menu-title="Governing low-code, no-code and vibe coding endeavors"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Governing low-code, no-code and vibe coding endeavors</h2> <p>Dealing with this situation is a <a href="https://www.techtarget.com/searchdatamanagement/feature/AI-data-governance-guidance-that-gets-you-to-the-finish-line">matter of governance</a>, said Thomas Squeo, CTO for the Americas at Thoughtworks, a technology consulting firm. But governance mechanisms should recognize the benefits of tools that better enable citizen development.</p> <p>"I don't think the answer is to lock these tools down, because the value is outsized relative to the risk," Squeo said.</p> <p>Vaskelė said many employees outside of IT have a passion for exploring tools and using them to save time on daily tasks. Advising citizen developers on best practices works better than trying to control tool use, she added.</p> <p>"I think that shadow IT, or shadow systems, will exist because people will always build, will always experiment," she said. "The best approach is to guide them and to define the boundaries."</p> <p>At Lithuania-based FL Technics, policies determine those boundaries, while employee training provides guidance on proper tool use, Vaskelė said. Employees are required to use tools within the corporate account. New tools need IT approval and a vendor cybersecurity assessment.</p> <p>Jason Brucker, a managing director at consulting firm Protiviti, said low-code, no-code and AI platforms simplify data integration and support process modernization, but raise governance and maintainability concerns.&nbsp; &nbsp;</p> <p>As a result, some organizations might be tempted to prevent the use of low-code/no-code data integrations outside specific IT teams, Brucker noted. But doing so does not guarantee success and can lead users to circumvent internal policies and procedures, he said. Strong governance, paired with data awareness and training, is a better approach, Brucker said, adding that well-designed governance initiatives <a href="https://www.techtarget.com/searchdatamanagement/opinion/Most-data-governance-wasnt-built-for-AI">establish clear guidance</a> and include validation expectations and monitoring.</p> </section> <section class="section main-article-chapter" data-menu-title="Boosting visibility, sticking to the basics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Boosting visibility, sticking to the basics</h2> <p>The balancing act of managing these tools while preserving development speed calls for comprehensive monitoring, particularly of key data pipelines.</p> <p>"Make the invisible, visible," Vaskelė said.</p> <p>To that end, enterprises should map their processes and prioritize those most critical to the business, she said. For FL Technics, examples include workflows in aircraft maintenance hangars. Vaskelė said that, in such cases, data lineage is crucial for showing where data flows and identifying its original source.</p> <p>Visibility into new tools <a href="https://www.computerweekly.com/news/366640697/Why-OpenClaw-agents-are-the-next-big-enterprise-challenge">entering the market</a> at astonishing speed is also important.</p> <p>"It's almost like a biome where, all of a sudden, there's all this new life growing at every turn, and you don't know necessarily how you are going to control it," Squeo said.</p> <p>He said the goal is to be aware of the tools and categorize them: "These are the ones that are dangerous, these are the ones that are acceptable, and these are the ones that are unknown and need to be determined."</p> <p>That assessment is an ongoing activity for chief data officers and other leaders, Squeo added.</p> <p>On controls for these citizen development tools, Squeo emphasized the fundamentals: <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">zero-trust principles,</a> data loss prevention (DLP), and managing north-south and east-west traffic.</p> <p>"Those are just good practices to overlay, and that's a combination of network management and DLP with a good security posture," he said. "Verify that you are doing the basics."</p> <p>In addition, Squeo said some organizations deploy network proxies that provide baseline visibility into AI tool access and govern outbound data flows, especially when combined with DLP and network security policies. Proxies also help identify usage patterns, enforce access restrictions and <a href="https://www.techtarget.com/searchsecurity/tip/How-to-perform-a-data-risk-assessment-step-by-step">reduce obvious risk exposure.</a> However, he noted that because proxy servers operate primarily at the network layer, they lack visibility into the context, sensitivity or intent of data -- especially with encrypted traffic.</p> <p>Proxies improve detection but don't fully resolve <a href="https://www.techtarget.com/searchsecurity/tip/Shadow-code-The-hidden-threat-for-enterprise-IT">ungoverned data flows or shadow AI</a> architectures, Squeo said.</p> <p>Kazmi said Ralliant's approach to preventing issues with data pipelines and integration layers consists of three pillars:</p> <ul class="default-list"> <li><b>Ownership.</b> Treat critical data flows as products with accountable owners for quality, availability and use.</li> <li><b>Platform.</b> Standardize on governed environments with embedded validation, access controls and policies.</li> <li><b>Visibility. </b>Continuously monitor pipelines, understand dependencies and validate performance in real time.</li> </ul> <p>"But what organizations are learning, ourselves included, is that speed redistributes complexity," Kazmi said.</p> </section> <section class="section main-article-chapter" data-menu-title="Moving beyond vibe coding"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Moving beyond vibe coding</h2> <p>Yasmeen Ahmad, managing director of product management for Google Cloud's data and AI cloud platform, said low-code and no-code are firmly establishing themselves in the enterprise because AI and agent technologies remove some of the manual toil in data preparation and pipeline work.</p> <p>"This is one of the first use cases where we have seen almost instant ROI," she said, noting the historically labor-intensive work of software engineers, data practitioners and developers.</p> <p>"Think about data engineering pipelines: There are armies of humans across our largest customers building data pipelines," Ahmad said.</p> <p>Skilled data engineers, rather than business users, tend to adopt agentic AI tools to quickly vibe-code a data pipeline, she said. At Google, that is evolving toward intent-driven engineering: A data engineer states the intent of a pipeline, and Google's Gemini chatbot provides a plan. The engineer iterates on the plan, and the AI tool writes the code against the updated version.&nbsp;</p> <p><i>John Moore is a freelance writer who has covered business and technology topics for 40 years. He focuses on enterprise IT strategy, AI adoption, data management and partner ecosystems.</i></p> </section> Low-code/no-code and vibe coding require data leaders to shift from gatekeepers to architects of trust to maintain data integrity and keep pipelines running reliably. https://cdn.ttgtmedia.com/rms/onlineimages/code_g1127196618.jpg https://www.techtarget.com/searchdatamanagement/feature/Govern-citizen-development-to-avoid-data-pipeline-downtime Fri, 08 May 2026 09:53:00 GMT <p>Data governance tends to falter when policies exist, but no one owns the decisions behind them.</p> <p>Large enterprises, faced with <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">increasing regulation</a> and heightened customer and investor scrutiny, have raised the stakes for enterprise decision-making. Implementing a data governance framework establishes a governance operating model, but few enterprises maintain a team that actually administers the program. Policy implementation is often left to individual departments or handed to IT teams with strong technical skills but limited business and regulatory knowledge, leaving gaps in compliance.</p> <p>As AI moves from pilot projects into production, companies steadily recognize that <a href="https://www.techtarget.com/searchenterpriseai/feature/9-data-quality-issues-that-can-sideline-AI-projects">data quality and availability</a> are binding constraints, as poorly governed data produces poorly governed AI systems. Shadow AI <a href="https://www.cybersecuritydive.com/news/shadow-ai-security-risks-netskope/808860/" target="_blank" rel="noopener">further complicates governance</a> by shifting decisions -- if they are made at all -- to employees without relevant experience, authority or training.</p> <p>A framework alone cannot adapt to these dynamics. It's a human problem. The question for senior leadership isn't whether to govern but <i>who</i> governs, how authority is structured and what business value the function delivers.</p> <p>The purpose of a data governance team isn't merely policy or regulatory compliance. The greater goal is enhancing the organization's capacity to <a href="https://www.techtarget.com/searchbusinessanalytics/feature/Data-driven-decision-making-case-study-Indeed">act with confidence using data-informed decisions</a>. Teams that accomplish this operate with both authority and responsibility, guided by an executive sponsor who treats the role as a standing commitment.</p> <section class="section main-article-chapter" data-menu-title="Executive sponsorship vs. data governance leadership"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Executive sponsorship vs. data governance leadership</h2> <p>Executive sponsorship is often treated as a single milestone: secure it. But that's only the first step. Trying to keep that sponsor engaged is harder and more important.</p> <p>Data leaders trying to secure sponsorship must determine whether the sponsor will commit time and influence on advancing the governance initiative. The <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-responsibilities-now-belong-in-the-C-suite">executive sponsor</a> must understand the job is more involved than initially expected. The division of labor between an executive sponsor and a data governance leader is as follows.</p> <p><iframe title="Executive sponsor vs. data governance leader" aria-label="Table" id="datawrapper-chart-TBHZJ" src="https://datawrapper.dwcdn.net/TBHZJ/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="713" data-external="1"></iframe></p> <p> <script type="text/javascript">window.addEventListener("message",function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data["datawrapper-height"][t]+"px";r.style.height=d}}});</script> </p> </section> <section class="section main-article-chapter" data-menu-title="Functions and duties in the data governance model"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Functions and duties in the data governance model</h2> <p>The structure of a governance team depends on the vulnerabilities and risks it manages, not on a generic template. However, a compact set of roles recurs in mature programs.</p> <p><iframe title="Data governance team roles and responsibilities" aria-label="Table" id="datawrapper-chart-ceUwt" src="https://datawrapper.dwcdn.net/ceUwt/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="1014" data-external="1"></iframe> <script type="text/javascript">window.addEventListener("message",function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data["datawrapper-height"][t]+"px";r.style.height=d}}});</script> </p> </section> <section class="section main-article-chapter" data-menu-title="Measuring the value of governance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Measuring the value of governance</h2> <p>Governance rarely moves the needle on company performance directly, so measurement must be indirect but rigorous. Avoiding fines is not meaningful KPI; it is a baseline requirement. Useful metrics fall into three families.</p> <p><iframe title="The three families of data governance metrics" aria-label="Table" id="datawrapper-chart-T2K7K" src="https://datawrapper.dwcdn.net/T2K7K/1/" scrolling="no" frameborder="0" style="width: 0; min-width: 100% !important; border: none;" height="501" data-external="1"></iframe></p> <p> <script type="text/javascript">window.addEventListener("message",function(a){if(void 0!==a.data["datawrapper-height"]){var e=document.querySelectorAll("iframe");for(var t in a.data["datawrapper-height"])for(var r,i=0;r=e[i];i++)if(r.contentWindow===a.source){var d=a.data["datawrapper-height"][t]+"px";r.style.height=d}}});</script> </p> <p>Other measures -- often called "vanity metrics" -- track activity rather than value. Examples include the number of policies authored, council meetings held or training hours delivered.</p> <p>Program effectiveness also depends on clear workflows. Three elements distinguish programs that work from those that struggle:</p> <ol class="default-list"> <li><b>Explicit dispute resolution paths.</b> When business units <a href="https://www.techtarget.com/searchdatamanagement/tip/The-data-ownership-blind-spots-putting-organizations-at-risk">disagree about data ownership</a> or definitions, someone must be named the decision-maker. Everyone should know who that is in advance.</li> <li><b>Documented decision rights.</b> Record who makes decisions, who is consulted and who is informed. A simple RACI matrix should prevent <a href="https://www.techtarget.com/searcherp/tip/RACI-matrix-for-project-management-success-with-example">much of the political friction</a> that governance programs generate. Ensure the RACI matrix is always up to date.</li> <li><b>Regular cadence.</b> Not every organization will have the same meeting rhythm, which should match its business pace. Monthly stewardship forums, quarterly team reviews and annual strategy resets work for most enterprises. But some organizations change more quickly and need a different cadence. &nbsp;Mergers and acquisitions also temporarily change cadence. In any instance, maintain transparent communication regarding decisions.</li> </ol> <p><em>Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.</em></p> </section> Regulations, AI and jumbled implementation oversight weaken decision-making. A dedicated team can help bring structure, accountability and consistent outcomes. https://cdn.ttgtmedia.com/visuals/searchCRM/learning_center/CRM_article_032.jpg https://www.techtarget.com/searchdatamanagement/feature/Who-belongs-on-a-high-performance-data-governance-team Wed, 06 May 2026 16:00:00 GMT <p>Without effective <a href="https://www.techtarget.com/searchdatamanagement/definition/data-governance">data governance</a>, growing volumes of data in IT systems are likely to become a disorganized morass, limiting their potential use. The risk of data misuse also increases due to lax controls.</p> <p>Conversely, well-governed data is consistent and accessible across the enterprise, enabling better-informed business decisions and <a href="https://www.techtarget.com/searchbusinessanalytics/opinion/Why-the-rush-to-replace-dashboards-with-AI-is-a-mistake">more accurate analytics and AI applications.</a> Companies are also less likely to experience serious data breaches or <a href="https://www.techtarget.com/searchdatamanagement/feature/Top-3-data-privacy-challenges-and-how-to-address-them">data privacy issues</a>, reducing their exposure to regulatory compliance problems, legal liabilities and reputational damage.</p> <p>As a result, developing a data governance strategy is a <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-responsibilities-now-belong-in-the-C-suite">high-priority item on the C-suite agenda</a> in well-run companies. The chief data officer and other data leaders play central roles in that process, typically managing it and working closely with their business counterparts to create and then implement the governance strategy.</p> <p>Getting started with data governance is a big undertaking that commonly requires a substantial budget and <a href="https://www.techtarget.com/searchdatamanagement/feature/Which-data-management-costs-should-survive-budget-cuts">significant resource commitments</a>. It might be tempting to buy a strategy from a consultancy or a <a href="https://www.techtarget.com/searchdatamanagement/feature/15-top-data-governance-tools-to-know-about">data governance software vendor</a> that promises a packaged set of policies and tools. But for optimal alignment with your organization's business operations and processes, it's best to develop a tailored one in-house, using these seven steps.</p> <section class="section main-article-chapter" data-menu-title="1. Document existing data governance processes"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Document existing data governance processes</h2> <p>Your company likely already has some data governance processes that should be incorporated into a formal strategy or replaced with new ones. Various people manage and oversee corporate data -- database administrators, backup admins, data architects and data quality analysts, for example. Document this by creating a directory of data assets and a corresponding list of managers and staff who are responsible or accountable for data.</p> <p>Don't be surprised if this exercise reveals some sobering, even shocking, oversights and gaps. The existing informal approach might reflect a messy reality, but getting a picture of current processes sets the stage for establishing a more strategic data governance program.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Secure executive sponsorship for the data governance program"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Secure executive sponsorship for the data governance program</h2> <p>Enlist senior business executives to sponsor, fund and promote the governance program. Their buy-in and top-down influence are critical because effective data governance requires participation and cooperation by departments and business units across the enterprise. But how can you win executive support for an initiative that might not show clear bottom-line benefits, at least not right away?</p> <p>Invoking fear, uncertainty and doubt is the usual default method. Horror stories of inaccurate data leading to bad business decisions, or of fines for failing to comply with <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">data privacy and protection laws</a>, might be enough to convince business leaders to back a governance initiative. By itself, however, this defensive approach isn't the optimal way to secure long-term data governance commitments.</p> <p>Instead, combine it with a more forward-looking approach. Explain that data governance is largely informal now and that the company needs a framework with more defined processes. Emphasize that implementing one will not only help <a href="https://www.techtarget.com/searchdatamanagement/feature/Proactive-practices-for-data-quality-improvement">improve data quality</a> and meet regulatory requirements but also make the organization more functional and resilient.</p> <p>Also, address upfront an issue that often causes resentment among business stakeholders and users: the perception that data governance stifles creative uses of data. Data governance policies don't restrict innovation. It's quite the opposite: By creating a more reliable data foundation, effective governance enables new ideas to flourish while reducing the risk of improper data use.</p> <figure class="main-article-image full-col" data-img-fullsize="https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f.png"> <img data-src="https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f_mobile.png" class="lazy" data-srcset="https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f_mobile.png 960w,https://www.techtarget.com/rms/onlineImages/data_management-need_to_govern_data-f.png 1280w" alt="Visual that lists key reasons why organizations need a data governance program." height="283" width="560"> <figcaption> <i class="icon pictures" data-icon="z"></i>These are key reasons why organizations need to develop and implement an effective data governance strategy. </figcaption> <div class="main-article-image-enlarge"> <i class="icon" data-icon="w"></i> </div> </figure> </section> <section class="section main-article-chapter" data-menu-title="3. Improve data literacy and skills across the organization"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Improve data literacy and skills across the organization</h2> <p>End users who understand the potential value of data and how to use it effectively are more likely to recognize the need to protect data assets and prevent misuse. To foster that understanding across the enterprise, <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">develop training to improve data literacy and skills</a> as part of the governance strategy.</p> <p>Enhanced data literacy also helps raise up data governance efforts in another way. End users often create duplicate reports, dashboards, spreadsheets and even entire databases because they don't know how to find existing ones. A <a href="https://www.techtarget.com/searchbusinessanalytics/feature/How-business-leaders-can-make-a-data-literate-culture-stick">data-literate culture</a> is better equipped to discover and reuse such assets, increasing efficiency and consistency and reducing the risk of data errors. This, in turn, helps streamline governance tasks.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Create a virtual governance team at first, then formalize roles"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Create a virtual governance team at first, then formalize roles</h2> <p>It's too much to ask a company to reorganize upfront to improve data governance. Instead, start by <a href="https://www.techtarget.com/searchdatamanagement/feature/Who-belongs-on-a-high-performance-data-governance-team">constructing some organizational scaffolding</a> around the existing ad hoc data structures. Identify the key roles currently involved in governance processes and create a virtual team to improve coordination and collaboration.</p> <p>As governance becomes more formalized, new roles will emerge. A data governance manager or vice president commonly leads a team of data governance specialists who coordinate the program. Some business or IT workers will become data stewards, with direct responsibility for implementing governance policies in particular data sets. That can be a full-time or part-time role, depending on the organization's size and the complexity of its governance needs.</p> <p>Establishing a data governance council or committee is also a must. It typically includes the following members:</p> <ul class="default-list"> <li>Representatives from all departments and business units.</li> <li>IT, legal and compliance executives.</li> <li>Data stewards or others with data ownership responsibilities.</li> </ul> <p>The council sets data governance policies, creates common data standards, prioritizes governance projects and resolves data-related disputes, among other responsibilities. Having one ensures there's broad input on data governance controls and helps pave the way for enterprise-wide adoption of the governance strategy.</p> </section> <section class="section main-article-chapter" data-menu-title="5. Decide how to measure the governance program's effectiveness"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Decide how to measure the governance program's effectiveness</h2> <p>For a data governance program to gain and maintain support in an organization, it's crucial to measure its effectiveness -- and show how it benefits the company. But effective data governance might not tangibly affect business performance. It also isn't easy to calculate KPIs for reduced business risks, such as avoiding regulatory fines or reputational damage.</p> <p>Instead, identify <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more">key data governance metrics to track</a> and tie them to business benefits such as improved decision-making, optimized business processes and stronger privacy protections. For example, use metrics on data accuracy, completeness, consistency, timeliness and duplication to <a href="https://www.techtarget.com/searchdatamanagement/tip/6-dimensions-of-data-quality-boost-data-performance">monitor data quality levels and document improvements</a> that make data more reliable for analytics and AI applications.</p> <p>Metrics also help identify governance issues. Tracking how often users access data provides insight into whether it's being used effectively. Low usage might indicate a lack of awareness or accessibility. An increase in the overall number of analytics users is a marker of the governance program's success, but further training might be required if metrics also show new users are creating reports and dashboards that duplicate existing ones.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Prioritize data governance for new AI applications"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Prioritize data governance for new AI applications</h2> <p>Governing data for AI applications is now a key consideration for data leaders and governance teams. Data readiness is <a href="https://www.techtarget.com/searchdatamanagement/tip/Experts-share-practices-to-overcome-AI-data-readiness">critical to successful deployments</a> of machine learning, generative AI and agentic AI tools. Effective data governance ensures that AI models are built on a solid foundation of high-quality data and don't use it in ways that violate privacy and ethics policies.</p> <p>For example, retrieval-augmented generation (RAG) frameworks pose specific data governance challenges in enterprise AI applications. RAG enables large language models (LLMs) to <a href="https://www.techtarget.com/searchenterpriseai/opinion/How-RAG-unlocks-the-power-of-enterprise-data">directly draw from enterprise data</a> -- it retrieves relevant documents or records from internal knowledge bases and uses them to generate responses to user queries.</p> <p>From a governance perspective, successful RAG use requires not only accurate, up-to-date data but also <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">data lineage documentation</a> that traces an LLM's output back to the original data sources for explainability and performance auditing. Well-managed access control is also necessary. Define and enforce user permissions in the RAG framework to prevent end users from inadvertently seeing data they can't access in conventional analytics applications.</p> <p>In addition to incorporating AI-related governance processes into your data governance strategy, <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-and-AI-governance-must-team-up-for-AI-to-succeed">align them with an AI governance program</a> that monitors and controls AI deployments more broadly. Data governance and AI governance are separate functions, but they go hand in hand and should be tightly integrated.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Select technologies that fit the data governance strategy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Select technologies that fit the data governance strategy</h2> <p>Various technologies can be used in data governance initiatives. Data governance software automates program management tasks, such as policy development, process documentation, data classification and workflow management. Data catalogs provide a <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">unified inventory of data assets</a>, with built-in governance, data lineage and data curation features. Analytics catalogs help users find relevant dashboards, reports and data visualizations and provide guidance on how to use them appropriately. AI tools are now being deployed to monitor data use and automatically enforce governance policies.</p> <p>For data processing, newer data lakehouse architectures combine the raw data storage of a data lake with the structured, governed repository of a data warehouse. Collapsing the separation between those two platforms streamlines data management and governance work and provides a single system that supports BI, advanced analytics and AI applications.</p> <p>But don't build a data governance strategy around specific technologies. Selecting tools that align with the strategy and support its goals will put the governance program on track to deliver the expected benefits, rather than running into a technology dead end that undermines the program.</p> </section> <section class="section main-article-chapter" data-menu-title="Move your organization forward with data governance"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Move your organization forward with data governance</h2> <p>The value of a pragmatic data governance strategy should steadily grow over time. A well-designed data governance process empowers business and analytics teams to do more with data, fostering a more data-driven and insightful organization. Rather than being seen as a restrictive discipline imposed on the organization, effective governance will be embraced as a strategic foundation that ensures data is treated as a critical business asset. Ultimately, it helps enhance decision-making, optimize business operations and enable the company to gain a competitive advantage over less data-centric rivals.</p> <p><b>Editor's note:</b> <i>This article was updated in April 2026 for timeliness and to add new information.</i></p> <p><em>Donald Farmer is a data strategist with 30-plus years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups.</em></p> </section> A strong data governance strategy enables more effective data use and helps prevent financial, legal and reputational problems. Follow these steps to develop one. https://cdn.ttgtmedia.com/rms/onlineimages/legal_g1169668297.jpg https://www.techtarget.com/searchdatamanagement/tip/6-key-steps-to-develop-a-data-governance-strategy Fri, 24 Apr 2026 17:01:00 GMT <p>Even with a governance program in place, organizations can still fall short.</p> <p>Lack of data ownership is a widespread issue. Many organizations are finding that vast amounts of data lack an assigned owner. According to the <a target="_blank" href="https://cpl.thalesgroup.com/about-us/newsroom/ai-the-new-insider-threat-facing-organizations" rel="noopener">2026 Data Threat Report</a> from global technology company Thales, only 34% of organizations know where all their data is stored, and only 39% can fully classify their data.</p> <p>Executives must identify unowned datasets within their organizations and assign ownership to ensure enterprise data is appropriately governed and secured. Not doing so puts the organization at financial and reputational risk.</p> <section class="section main-article-chapter" data-menu-title="Common data ownership oversights"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Common data ownership oversights</h2> <p>Quality data has become a critical asset for organizations, especially for <a href="https://www.techtarget.com/searchenterpriseai/feature/9-data-quality-issues-that-can-sideline-AI-projects">automation, analytics and AI</a>. Data owners, data stewards and chief data officers are responsible for ensuring that data meets established standards and <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-lineage-documentation-imperative-to-data-quality">has trackable lineage</a>.</p> <p>But there are still types of data that organizations fail to capture and govern effectively, including:</p> <ul class="default-list"> <li><b>Unstructured content from communication channels.</b> Transcripts from messaging and collaboration apps, meeting transcripts and recordings, and content from emails and social media exchanges all lack ownership but might contain critical business information.</li> <li><b>Data generated by shadow IT systems.</b> About 70% of CIOs believe that the business units in their organizations deploy unsanctioned tech, according to Flexera's <a href="https://www.flexera.com/resources/reports/ITV-REPORT-IT-Priorities" target="_blank" rel="noopener">2026 IT Priorities Report</a>. That creates shadow data -- that is, data generated and stored outside the purview of the organization's IT and security controls as well as its data governance program.</li> <li><b>Data developed in sandboxes and for temporary projects.</b> This spans from developmental databases and test environments to intermediate datasets to <a href="https://www.techtarget.com/searchenterpriseai/tip/Explore-the-role-of-training-data-in-AI-and-machine-learning">training data for AI and machine learning</a> models.</li> <li><b>Siloed data.</b> Many data types fall into this category, including data stored in individual desktop folders, temporary reports, disconnected spreadsheets and data in legacy systems. Some also put vendor and third-party data in this category. Dark data also falls into this category, even though the business no longer uses it.</li> <li><b>Operational and machine data.</b> Various data types belong here, ranging from systems logs, IoT sensor data and API payloads to metadata. These datasets often lack a single owner and are overlooked in management and governance.</li> </ul> </section> <section class="section main-article-chapter" data-menu-title="Unowned data governance challenges"> <h2 class="section-title"><i class="icon" data-icon="1"></i>Unowned data governance challenges</h2> <p>Establishing governance and ownership over unowned data is challenging. Experts noted that these data types are hard to classify, easy to overlook and span multiple business functions. Consequently, no single business leader claims or is assigned ownership. Or it might be that data ownership lies with many or all leaders, which lacks accountability.</p> <p>Regardless, unowned data puts the organization at risk of costly data breaches, <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">regulatory violations</a> and inaccurate outputs from analytics, automation and AI initiatives due to poor data quality.</p> <p>With such consequences in mind, many executives seek to improve their organization's overall data management and governance. According to Workvia's <a target="_blank" href="https://www.workiva.com/sites/workiva/files/pdfs/workiva-2026-exec-benchmark-survey-en.pdf" rel="noopener">2026 Executive Benchmark Survey</a>, business leaders ranked "strengthening data governance" as the second-highest priority for digital transformation projects, behind automating data collection and validation.</p> <p>Similarly, Informatica's <a href="https://www.informatica.com/about-us/news/news-releases/2026/01/20260127-new-global-cdo-report-reveals-data-governance-and-ai-literacy-as-key-accelerators-in-ai-adoption.html" target="_blank" rel="noopener">CDO Insights 2026 Report</a> found that 86% of 600 surveyed data leaders planned to increase data management investments -- and 41% seek to boost data and AI governance. This is in response to concerns about poor data quality affecting business objectives.</p> <p>Ultimately, data management and governance won't succeed if they can't be accounted for. Assigning ownership to unowned or over-shared data will help organizations strengthen data governance and boost data quality -- and their own trustworthiness.</p> <p><em>Mary K. Pratt is an award-winning freelance journalist with a focus on covering enterprise IT and cybersecurity management.</em></p> </section> Organizations can't claim to have good data governance when they still have unowned data. Assigning ownership to siloed and dark data is critical to enterprise success. https://cdn.ttgtmedia.com/rms/onlineimages/folder-files07.jpg https://www.techtarget.com/searchdatamanagement/tip/The-data-ownership-blind-spots-putting-organizations-at-risk Fri, 24 Apr 2026 14:20:00 GMT <p>Data is one of an organization's most valuable assets. But without a comprehensive data strategy as a foundation, it often becomes fragmented, inconsistent and difficult to access or trust for business decision-making.</p> <p>An effective enterprise data strategy establishes a structured approach to managing, governing and using data in alignment with business objectives. That enables companies to <a href="https://www.techtarget.com/searchdatamanagement/opinion/Turning-data-into-a-strategic-advantage">unlock greater value from their data assets</a> through improved decision-making, optimized business processes and increased operational efficiency. It also helps them boost innovation and gain a sustainable competitive advantage over less data-driven rivals.</p> <p>The data strategy shouldn't focus on implementing new technologies. That comes later, driven by the strategy. Instead, it should set the direction for data management processes, address common data-related challenges and build the capabilities needed to <a href="https://www.techtarget.com/searchdatamanagement/opinion/Data-management-and-governance-key-to-successful-AI-use">support planned data use</a> across the enterprise.</p> <p>Follow these 12 steps to develop a data strategy that accomplishes those things and positions your organization to realize long-term business benefits.</p> <section class="section main-article-chapter" data-menu-title="1. Define clear business objectives for data initiatives"> <h2 class="section-title"><i class="icon" data-icon="1"></i>1. Define clear business objectives for data initiatives</h2> <p>A successful enterprise data strategy is grounded in close alignment between data initiatives and business goals. Data management and analytics efforts should directly support priorities such as enabling better-informed decision-making, enhancing customer experience, improving business operations, <a href="https://www.techtarget.com/searchdatamanagement/feature/Modern-data-architectures-as-a-risk-management-strategy">reducing risks</a> and fostering innovation.</p> <p>To achieve this alignment, work closely with senior executives and business managers to identify critical objectives that depend on effective data use. Engaging with key stakeholders at the outset ensures the data strategy addresses real business needs and guides appropriate technology choices to help meet them. Data initiatives tied to measurable business outcomes are more likely to gain executive support and sustained investment in the resources required for long-term success.</p> </section> <section class="section main-article-chapter" data-menu-title="2. Assess the existing data landscape in your organization"> <h2 class="section-title"><i class="icon" data-icon="1"></i>2. Assess the existing data landscape in your organization</h2> <p>Next, get a complete understanding of the organization's current data environment. A comprehensive assessment documents existing technologies, capabilities, challenges and opportunities for improvement. The data management team should conduct it with clear visibility across data domains and business processes enterprise-wide.</p> <p>As part of the assessment, review source systems, data platforms, integration processes, governance structures and analytics applications, as well as how data flows between IT systems in different departments or business units. This uncovers issues such as data silos, inconsistent data definitions, limited metadata visibility and restricted access to relevant data. Identifying these gaps enables data leaders to prioritize initiatives and create a realistic roadmap for implementing the data strategy.</p> </section> <section class="section main-article-chapter" data-menu-title="3. Specify the desired state for data management and analytics"> <h2 class="section-title"><i class="icon" data-icon="1"></i>3. Specify the desired state for data management and analytics</h2> <p>Once the current-state assessment is complete and the results have been evaluated, articulate what works well and where changes are needed in data management and analytics processes. Defining the desired state clarifies what the organization can achieve through those changes. This vision should be based on the previously identified business imperatives for each data domain and function.</p> <p>As part of this step, set data quality expectations and outline plans to harmonize core data management processes, such as data integration, metadata management and master data management. Doing so ensures consistency across systems and reliable access to <a href="https://www.techtarget.com/searchdatamanagement/opinion/Trusted-data-is-the-foundation-of-data-driven-decisions-GenAI">relevant and trustworthy data</a>.</p> </section> <section class="section main-article-chapter" data-menu-title="4. Identify and prioritize critical data domains"> <h2 class="section-title"><i class="icon" data-icon="1"></i>4. Identify and prioritize critical data domains</h2> <p>The strategic value of data varies. While an enterprise data strategy by definition should address all data domains, focus initial implementation efforts on the domains and associated data sets that are most critical to business operations and decision-making.</p> <p>Identifying and prioritizing the highest‑value data domains enables data leaders to direct resources to areas where data management and analytics improvements will have the greatest business impact. In a retailer, for example, improving customer data quality enables more accurate analytics for targeted marketing and better customer service. Focusing on high‑value areas also helps demonstrate the data strategy's value and build momentum toward a more <a href="https://www.techtarget.com/searchdatamanagement/tip/Use-these-steps-to-successfully-build-your-data-culture">data‑centric culture</a>.</p> </section> <section class="section main-article-chapter" data-menu-title="5. Create an implementation roadmap"> <h2 class="section-title"><i class="icon" data-icon="1"></i>5. Create an implementation roadmap</h2> <p>After defining what your organization aims to achieve with data to support business priorities and what's required to do so, create an implementation roadmap that details how it will get there. A well-designed roadmap sequences data initiatives over time in a way that's achievable and measurable.</p> <p>That requires balancing ambition with realism to enable sustained, disciplined progress on the data strategy rather than a series of disconnected short-term projects -- or, worse, overpromising on planned deployments. The roadmap should also connect long-term goals, such as becoming more data-driven or AI-enabled, to concrete steps across data management and analytics processes.</p> </section> <section class="section main-article-chapter" data-menu-title="6. Develop data principles and strategic guardrails"> <h2 class="section-title"><i class="icon" data-icon="1"></i>6. Develop data principles and strategic guardrails</h2> <p>Incorporate data principles and strategic guardrails into the data strategy so they actively shape decisions on data management and use, rather than being abstract guidelines. Foundational principles -- such as treating data as an enterprise asset, ensuring it's accurate and accessible, and establishing a single source of truth through transparent data management practices -- should directly inform the data operating model and architecture. This drives data consistency, reuse and trust across the organization.</p> <p>Strategic guardrails are operational constraints and requirements in areas such as privacy, security, <a href="https://www.techtarget.com/searchbusinessanalytics/feature/Why-ethical-use-of-data-is-so-important-to-enterprises">ethical data use</a>, data quality and data platform design. Embed them in the data strategy as part of data governance policies and the implementation roadmap. Aligning suitable guardrails with the execution of data initiatives provides clear direction on appropriate data use, reduces data-related risks and enables BI, data science and business teams to innovate confidently within well-defined boundaries.</p> </section> <section class="section main-article-chapter" data-menu-title="7. Build a data governance framework and assign data ownership"> <h2 class="section-title"><i class="icon" data-icon="1"></i>7. Build a data governance framework and assign data ownership</h2> <p>A strong data governance program is a <a href="https://www.techtarget.com/searchdatamanagement/tip/6-key-components-of-a-successful-data-strategy">critical component of a data strategy</a>. Effective data governance ensures that data remains consistent and reliable and that it's managed and used properly. Without it, various problems can arise. For example, different departments might create conflicting data definitions or data quality might deteriorate, compromising business decisions due to incomplete or inaccurate information.</p> <p>Include <a href="https://www.techtarget.com/searchdatamanagement/tip/5-benefits-of-building-a-strong-data-governance-strategy">implementing the data governance framework</a> as a foundational item in the data strategy's roadmap. The strategy should also detail expectations for managing data throughout its lifecycle and the role of data governance in supporting business objectives. Additionally, work with business stakeholders to assign ownership of data assets to appropriate individuals or teams and task them with ensuring the data they oversee is managed and used in accordance with governance policies.</p> </section> <section class="section main-article-chapter" data-menu-title="8. Design an enterprise data architecture"> <h2 class="section-title"><i class="icon" data-icon="1"></i>8. Design an enterprise data architecture</h2> <p>A <a href="https://www.techtarget.com/searchdatamanagement/definition/What-is-data-architecture-A-data-management-blueprint">data architecture</a> provides the technical foundation for managing and delivering data. It defines and visualizes how data is processed, integrated, stored and accessed across systems. However, in many organizations, the existing data architecture has been developed over time, often in a piecemeal fashion without an enterprise-wide focus. As a result, redundancies and gaps in the architecture create challenges with data access and use.</p> <p>To address these issues, design an enterprise data architecture as part of the data strategy. In addition to a high-level architectural blueprint, it should include artifacts such as data models, data flow diagrams and documents that map data use to business processes. A well-designed data architecture guides data management processes, helps teams identify data challenges and supports both operational reporting and advanced analytics.</p> </section> <section class="section main-article-chapter" data-menu-title="9. Implement security, privacy and regulatory compliance controls"> <h2 class="section-title"><i class="icon" data-icon="1"></i>9. Implement security, privacy and regulatory compliance controls</h2> <p>Protecting the ever-increasing volumes of data that organizations collect and use is critical to avoiding business problems. In addition to strategic guardrails that set high-level boundaries on data management and use, a data strategy must include specific controls to <a href="https://www.techtarget.com/searchdatamanagement/feature/Top-3-data-privacy-challenges-and-how-to-address-them">mitigate data security and privacy risks</a>. For example, ensure that only authorized users can access sensitive data and that potential security threats can be detected and addressed quickly through predefined incident response plans.</p> <p>Regulatory compliance is also a broader issue now due to the <a href="https://www.techtarget.com/searchdatamanagement/tip/Data-governance-regulations-that-executives-should-know">growing number of data protection laws</a> that require responsible management of personal information and transparency about how data is used. Integrate compliance mechanisms into the data strategy to help reduce legal risks and maintain trust with customers and business partners.</p> </section> <section class="section main-article-chapter" data-menu-title="10. Enable data accessibility and increased data literacy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>10. Enable data accessibility and increased data literacy</h2> <p>Making trusted data accessible to the people who need it is a core objective of an enterprise data strategy. Data access is no longer restricted to technical specialists. A modern data strategy supports controlled, governed access for business users and data analysts through user-friendly dashboards, self-service analytics tools and <a href="https://www.techtarget.com/searchdatamanagement/answer/What-steps-are-key-to-building-a-data-catalog">centralized data catalogs</a>.</p> <p>However, data accessibility alone isn't enough. Increased data literacy is also required across the organization to maximize the business value derived from data assets. As part of the data strategy, <a href="https://www.techtarget.com/searchbusinessanalytics/tip/Develop-a-data-literacy-program-to-fit-your-company-needs">develop a data literacy program</a> that sets expectations for workers and includes training to help them become more data-literate.</p> </section> <section class="section main-article-chapter" data-menu-title="11. Build in support for BI, advanced analytics and AI applications"> <h2 class="section-title"><i class="icon" data-icon="1"></i>11. Build in support for BI, advanced analytics and AI applications</h2> <p>In the past, data strategies often focused primarily on delivering data for use in BI and reporting applications. But now they must also focus on the data needed for expanding deployments of advanced analytics and AI applications in companies.</p> <p>Build support for techniques and tools such as predictive analytics, machine learning and both generative AI and agentic AI into the enterprise data strategy. Used effectively, they help organizations identify patterns in large data sets, forecast trends, explore data more efficiently and optimize or automate business processes. However, they <a href="https://www.techtarget.com/searchdatamanagement/opinion/The-future-of-AI-depends-on-better-data-not-bigger-models">depend on the strong data foundation</a> that a data strategy provides.</p> </section> <section class="section main-article-chapter" data-menu-title="12. Define metrics to track and evolve the data strategy"> <h2 class="section-title"><i class="icon" data-icon="1"></i>12. Define metrics to track and evolve the data strategy</h2> <p>A data strategy should evolve over time as business priorities, data sets, technologies and regulations change -- and as problems are identified. To guide this evolution, define <a href="https://www.techtarget.com/searchdatamanagement/feature/Data-governance-metrics-Data-quality-data-literacy-and-more">KPIs and other metrics</a> to track the effectiveness of data initiatives. Include ones on things such as data quality, governance activities and data availability, security and use.</p> <p>Monitoring them enables data and business leaders to evaluate progress on initiatives and identify areas for improvement. Regular reviews and continuous refinement of the data strategy ensure that it remains aligned with the organization's needs and continues to deliver business value. Spell out the need for that upfront, when setting expectations for the strategy, so it isn't a surprise to anyone.</p> <p><em>Anne Marie Smith, Ph.D., is an information management professional and consultant with broad experience across industries. She has also designed and delivered numerous data management courses and educational programs.</em></p> </section> Here are 12 to-do items for data leaders developing a data strategy to help their organization use data more effectively for analytics and business decision-making. https://cdn.ttgtmedia.com/visuals/search400/iseries_database_manage/search400_article_014.jpg https://www.techtarget.com/searchdatamanagement/tip/Developing-an-enterprise-data-strategy-10-steps-to-take Fri, 17 Apr 2026 15:50:00 GMT 60 webmaster@techtarget.com