Lakebase: Another vendor buzzword or a useful new technology?

Common lakebase design principles

Although the Databricks and Zilliz lakebases focus on different use cases, they share key design principles.

Databricks Lakebase stems from the company's 2025 acquisition of Neon, a startup that offered a version of PostgreSQL, the most widely used open source transactional database. The revised technology is now built the way most cloud data platforms are: compute is separated from storage, resulting in a serverless database engine. Because Lakebase's PostgreSQL engine doesn't store any data, it can start in seconds, scale up or down while running, stop when idle, and make a working copy of an entire database almost instantly.

Zilliz Vector Lakebase applies the same idea to vector data, the numerical representation that AI models use to search documents, images and other unstructured data for relevant information. Search indexes in Milvus 3.0 are built directly over the files in a data lake without storing a copy of them. Because accessing cloud object storage is slower than accessing the disks inside a monolithic database server, Zilliz has also added a new Loon storage engine that optimizes S3 layouts to reduce the amount of data that must be read to answer individual queries.

The ultimate shared design principle is enabling users to remove the data pipelines between transactional systems and analytical data repositories, eliminating one of the biggest data management challenges for decades: building and running them.

For example, a financial services firm might build a fraud detection model in a data lake or lakehouse to score every customer. But the fraud risk scores are most useful inside the application that approves or blocks transactions. Conventionally, a data engineer would build a pipeline to move the scores into the application, and another to load the resulting transactions into the lakehouse for the next model training run and other analytics uses. Each pipeline adds delay and cost and, too often, is something else that can break in a data architecture. If the application and the analytics instead use a single copy of the data in a shared repository, both pipelines can go away.

Questions and issues for data leaders to consider

In data and analytics applications, human workloads are generally quite steady: people log on in the morning and use systems at a human pace. Agentic AI workloads are not so steady: an agent might be idle for hours and then need a large amount of compute resources for a few minutes.

Agents also need data that provides the context, constraints and accountability to properly guide their decisions and actions. For example, an AI agent tasked with approving a refund requires the customer's current orders, their fraud risk score and the company's refund policy documents -- the latter represented as vectors -- to all be delivered at the same time.

When thinking about the potential value of a lakebase, one issue to consider is whether the agent can wait for traditional pipelines to feed it that data. If the pipelines don't work smoothly, the refund process could be delayed or even fail.

But we are in the early days of this new technology, and there is more to consider when evaluating it. Asking the following questions can help data leaders decide whether a lakebase is the right platform for their organization and for what workloads:

• Who initiates the workload? If human users are running an application with steady traffic, a conventional database is reliable and convenient, and probably less expensive than a lakebase. But if the users are numerous AI agents from different development efforts, with uneven and often high demand for data and compute resources, the elastic scalability offered by a lakebase could be beneficial.
• How much latency is acceptable? Applications such as payment processing, financial trading and high-concurrency ticketing require predictable response times of a few milliseconds, which object storage cannot promise. For those uses, a well-tuned OLTP system that stores data on local disks in a database will still perform better. However, most internal applications and current agent workflows only require response times in the tens of milliseconds. A lakebase might be good enough in those cases.
• What platforms do data teams already use? For teams on Databricks or Microsoft Fabric, which interoperates with an Azure version of Databricks Lakebase, a single platform for both operational and analytical workloads eliminates a layer of integration work and makes it easier to enforce platform standards under one catalog. For other teams, this is a platform decision, not just a database vs. lakebase one.

A buzzword, but more than that

So, is lakebase a buzzword or a breakthrough technology? Both.

Yes, it is a buzzword that puts a marketing term to an existing trend -- the separation of compute from storage at scale -- and ties it to a specific platform. But that doesn't mean there isn't a useful technology behind it -- not as a replacement for high-throughput transaction systems, but for agentic AI workloads, online feature stores and applications built on data lakes and lakehouses. That is roughly where the lakehouse was six years ago, in a different use case scenario.

One thing that will certainly change over time is the definition of acceptable latency. In the future, when AI agents are the primary users of data, "fast enough" will need to be redefined -- and data platforms will need to evolve accordingly. The lakebase is an early answer to that question.

Donald Farmer is a data strategist with 30+ years of experience, including as a product team leader at Microsoft and Qlik. He advises global clients on data, analytics, AI and innovation strategy, with expertise spanning from tech giants to startups. He lives in an experimental woodland home near Seattle.