Metadata management has long been a largely unheralded task for data teams. But it’s taking on new prominence — and urgency — as metadata becomes a critical component of efforts to give AI systems the context they need to generate accurate results.
Metadata is now also vital for setting guardrails that prevent AI from improperly accessing or misusing data. Both uses make effective metadata management a top priority for data leaders in the AI era, according to consultants and data professionals.
“Metadata is finally getting its due,” said Stephanie Paradis, a data governance consultant at First San Francisco Partners. In a session during Dataversity’s recent AI Governance Online conference, Paradis added that she expects metadata to become a competitive differentiator in AI applications for organizations that manage it well.
To achieve that, data teams must evolve from passively documenting metadata to actively managing it as an information resource that provides an interpretation and control layer for AI, said Paradis and Gretchen Burnham, another First San Francisco consultant who gave a joint presentation with her. “The value of metadata is in its usage, not its collection — and that has never been more true than it is right now,” Burnham said.
“The value of metadata is in its usage, not its collection — and that has never been more true than it is right now.”
Gretchen Burnham
Consultant, First San Francisco Partners
In a separate panel discussion during the conference, Anant Somvanshi, a senior specialist in the Data & AI Defense Office at financial services firm Vanguard, similarly said he’s seeing a shift in his organization and others from static documentation to actionable metadata that supports data discovery by AI tools. The metadata collected in data catalogs now also helps drive data access controls for AI agents, AI traceability and other governance and security functions, Somvanshi said.
A dual approach to managing metadata for AI
Paradis and Burnham said two types of machine-readable metadata are required: semantic metadata that provides the context for AI to understand data, and operational metadata that specifies the data AI tools can access and what they’re permitted to do with it.
Organizations have used metadata to create semantic layers for nearly 20 years, Burnham said. But until now, those efforts primarily focused on BI applications for human end users. By contrast, the semantic metadata AI needs includes not only data definitions, lineage documentation and business rules but also the knowledge of business processes that people have gained through training and experience.
Without all that information, AI models and agents are likely to make errors when analyzing data or responding to queries — and while humans will often notice that something doesn’t look right in the data they’re using, Paradis said AI typically produces a confident, plausible-sounding answer instead of an error message.
Similarly, people can use their judgment to avoid missteps when using data, based on organizational policies and norms. But such controls “are completely absent when the actor is an AI agent,” Paradis said, noting that agents are inherently designed to take full advantage of whatever permissions they have. She said operational metadata must provide a safety net to prevent data security and privacy issues in AI applications — for example, by supporting narrow permissions for specific tasks.
AI stakes, metadata challenges create a high bar
In addition to overseeing metadata collection, curation and stewardship, data governance leaders should orchestrate the overall management process with business, development, security and compliance teams, Burnham and Paradis said. On the plus side, that makes their role even more important as organizations increase AI deployments. But it also raises the bar for data teams, as Paradis put it.
Metadata management remains inherently challenging, as well. Anne Marie Smith, principal consultant at Alabama Yankee Systems, told TechTarget that metadata is often inaccurate, incomplete, inconsistent, outdated or siloed. Such issues can be compounded by unclear data ownership, limited business engagement and difficulty maintaining metadata as business requirements change, she added.
While those have always been challenges, Smith said AI further amplifies them because of its need for reliable metadata to interpret data correctly and generate trustworthy outputs.
Individual databases and data warehouses, lakes and lakehouses commonly offer capabilities for managing their own metadata. But Kevin Petrie, a data management analyst at BARC U.S., said enterprise data catalogs or broader data intelligence platforms are needed to organize metadata across systems. If an organization has multiple catalogs, Petrie said data teams should integrate metadata across them to give both AI tools and human users a cohesive view of enterprise data assets.
Burnham, Paradis, Petrie and Smith all agreed that involving business leaders in the metadata management process and building it around specific AI use cases are critical to its success. Doing so helps data leaders prioritize work and avoid having their teams spend valuable time and energy managing metadata that has limited business value, Petrie said.
On the one hand, Burnham said AI requires comprehensive sets of metadata — “every scrap” of it that’s relevant to a particular application. On the other hand, that doesn’t mean data teams should collect “absolutely everything and make this huge museum of metadata,” she said. Tying the process to planned AI uses will target metadata requirements and help justify the necessary management and governance investments, Burnham added.
Craig Stedman is a senior reporter on the TechTarget Data news team, covering analytics, data management and data backup topics. He has more than 40 years of experience as an enterprise IT writer and editor.
