Integrate structured and unstructured data for better AI outputs

What integration looks like in practice

Multiple organizations have successfully combined structured and unstructured data for agentic AI. Two specific public examples can be found in the healthcare and insurance verticals. These examples show what the effort to integrate structured and unstructured data buys: time saved in one industry and money saved in the other.

Healthcare

Healthcare organizations have an overwhelming volume of data to manage. New Jersey-based healthcare provider PromptCare uses an AI agent to review unstructured medical records and assess whether a patient meets therapy qualification criteria.

That data is measured against structured qualification criteria for a specific patient before a person makes the final call. According to PromptCare CIO Phil Merrell, the approach cut referral processing time from 15 days to four.

Insurance

Shift Technology's fraud-detection platform reads adjuster and investigator unstructured notes alongside structured claim fields such as policy dates and amounts. A company case study describes the platform linking a $525,000 kitchen-fire claim to a similar claim from two years earlier, a match that structured data alone would have missed.

How to integrate structured and unstructured data for AI

While the benefit of using both unstructured and structured data is obvious, it's not quite as obvious how to use both types of data together. By definition, unstructured data has no structure, so it won't fit into the databases, tables and rows of structured data.

That doesn't mean, however, that there aren't solid best practices for combining the two types of data. IT leaders can structure data pipelines in a way that moves unstructured content out of file shares and email threads and into the same answer an agent already pulls from structured records.

Use the following four practices to turn scattered data into a single retrieval process:

1. Consolidate storage in a unified platform. Using platforms such as a data lakehouse, which combines a data warehouse's governance and query performance with a data lake's flexibility for unstructured content, can make both data types equally accessible to agentic AI.
2. Classify and tag content at ingestion. File names and timestamps are not enough to match unstructured content to the structured records it relates to. Vendor Databricks provides guidance on building unstructured data pipelines, recommending the use of content-based metadata such as topics and named entities. In that approach, an unstructured data document can be linked to the account, claim or patient record it describes.
3. Build one retrieval layer instead of two silos. When it comes to actual retrieval, structured and unstructured data are often queried by separate systems. That approach doesn't connect the two data types. Having some kind of semantic graph, so that a single query pulls matching records and documents together, is a critical step. A semantic layer turns the tags added in the above step into connections across systems, rather than labels sitting on individual documents.
4. Match the query to the data type. Inside that same retrieval layer, precise questions and open-ended questions still call for different handling. A Google study comparing agent retrieval methods found a semantic, metadata-driven agent achieved 65.7% higher precision retrieving usable data than a baseline agent searching open, unstructured content. That said, the baseline agent answered 40% more questions by casting a wider net. The best approach is to send exact, execution-critical lookups to the structured side and pull only the most relevant handful of unstructured chunks for everything else.

Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.