AI pressures expose outdated data pipeline architecture

What AI workloads require from data pipelines

Five years ago, many enterprise data pipeline architectures were designed mainly around analytics refresh cycles. The target was usually a data warehouse or a BI dashboard. BI and analytics can tolerate some latency, and humans can make adjustments -- such as data analysts performing reviews and addressing issues -- to get the right results before they're used to influence business decisions.

Autonomous AI agents have a different risk profile. AI agents make concurrent, multi-step data requests and can act on outputs with limited -- or no -- human review. As a result, a stale or malformed data input can propagate directly into AI model behavior or business decisions.

In a March 2026 blog post, IDC said it expects 80% of agentic AI use cases to require real-time, contextual and widely accessible data. The post also said the market research firm expects 40% of the top 2,000 public companies worldwide to adopt modern event streaming technology and prebuilt real-time data views to support AI agents by 2027.

That shift changes the requirements for a data pipeline architecture:

Data ingestion and streaming

With traditional batch ingestion, data can be delivered too slowly for real-time analytics, agentic workflows or automated decision support, and schema changes can break workflows without warning.

To fill the gap, enterprises use event-driven architecture. It's a layered approach that includes several key steps.

• Change data capture. CDC converts database changes into a continuous stream of events at the source.
• Stream processing. Streaming ingests events continuously to handle stateful operations and help synchronize schema changes across systems.
• Validation at ingestion. The final check before data reaches any users, this covers schema enforcement, nullability checks and range checks.

There are numerous public examples of organizations that have embraced this approach.

Netflix built a "real-time distributed graph" on an event-driven architecture when it realized batch processing could not meet the latency demands of real-time applications. Uber made a similar shift, replacing batch ingestion into a data lake with streaming -- a move that cut data latency from hours to minutes. Shopify runs stream processing pipelines at scale to deliver real-time recommendations and buyer signals across its platform.

Transformation and data quality

When an agent acts on the output of a poorly tested data transformation, there is no analyst to catch errors that were missed. A bad join or stale column definition can flow into model responses, retrieval results or automated decisions.

For AI workloads, the standard is combining version-controlled transformation logic, automated quality enforcement and data contracts to reduce the risk of errors reaching AI systems.  

• ELT vs. ETL. Many modern cloud data environments favor ELT, which loads raw data first and transforms it for different uses in the destination platform. ELT can support flexible transformation in cloud data warehouses and lakehouses to take advantage of scalable cloud compute resources. ETL remains useful when data must be standardized, filtered or governed before it lands -- especially when bandwidth is limited, storage costs are a concern or compliance requires data masking before loading.
• Version-controlled transformation logic. Data transformation tools, such as DBT, document transformations and make them testable and auditable, with a lineage graph from raw sources to final tables. For AI workloads, this lineage should also extend to features, embeddings, model training datasets and application-facing data products.
• Data contracts. These are versioned agreements between data producers and consumers covering schema, data types, quality SLAs and change workflows. PayPal has open sourced its data contract template as a reference implementation.

Orchestration and observability

Data orchestration ensures pipelines run on schedule, but a completed job does not prove that the output is correct. Data observability fills that gap, but most tooling was built for structured warehouse data and does not always extend to vector stores or the unstructured sources AI agents often rely on.

The required shift is from monitoring job execution to monitoring data quality, using several components.

• Pipeline orchestration. It manages scheduling, dependencies and failure recovery. Orchestration tools can handle complex pipelines that need to enforce data lineage and freshness policies.
• Quality-based alerting. This supports monitoring of data quality issues and events. Simply documenting that a job ran is not evidence that its output is correct.
• Data observability. Observability platforms monitor freshness, volume, schema and distribution using anomaly detection. Leading tools have extended this to unstructured data and vector databases.
• Graceful degradation. When a pipeline stage fails, serving last-known-good data and flagging partial results can help prevent incomplete or bad data from reaching downstream systems or end users. For higher-risk workflows, graceful degradation can also include quarantining suspect records, rolling back to previously validated outputs, triggering automated alerts and notifying affected users, systems or data consumers.

Access controls and data lineage

Data lineage has long been a foundational aspect of data pipelines, enabling users to understand where data comes from. The problem is that traditional lineage tooling often stops at the data warehouse boundary, leaving limited visibility of what data was embedded and stored in AI vector databases.

For agentic AI data pipelines, organizations need both access control and lineage that reach the AI data retrieval layer.

• Lineage tracking. Open standards let tools automatically capture and share data lineage. Column-level lineage makes it easier to see where data comes from, understand the impact of changes, and maintain audit trails.
• Retrieval-layer access controls. Role-based and attribute-based access should be enforced at the vector store, with PII detected and redacted during embedding.
• Audit logging. User queries and retrieved documents must be logged to create a traceable record of what data informed each model response.
• Governance frameworks. Frameworks such as the OWASP Top 10 for LLMs and the NIST AI Risk Management Framework provide guidance on identifying risks, implementing controls and managing AI systems responsibly.

What to do now

Batch pipelines still serve analytics workloads well, but many won't deliver the freshness, quality or traceability required for AI-driven applications.

Use the following readiness checks to identify where a pipeline needs an upgrade:

• Ingestion is ready when schema changes no longer require manual intervention.
• Transformation is ready when a bad join gets caught before it reaches a model.

• Observability is ready when alerts fire on data quality, not just job completion.
• Governance is ready when every retrieval event has an audit trail.

Sean Michael Kerner is an IT consultant, technology enthusiast and tinkerer. He has pulled Token Ring, configured NetWare and been known to compile his own Linux kernel. He consults with industry and media organizations on technology issues.