StarTree adding Iceberg support to simplify, speed analysis

New capabilities

Open tables are datasets that are compatible with numerous engines such as Apache Spark, Apache Flink, Presto/Trino and ClickHouse as well as any cloud data platform that provides support for open tables.

Due to their flexibility to work with various systems, they are gaining popularity among both enterprises that store data in multiple systems and users put off by potential vendor lock-in when using a proprietary table format that only works with one provider's data management platform.

Meanwhile, Iceberg has emerged as perhaps the most popular open table format.

Beyond being vendor agnostic, Iceberg provides lakehouse capabilities that enable users to combine structured and unstructured data, separates metadata from file storage to improve workload performance and has a dedicated community of users such as AWS, Apple and Netflix -- which founded Iceberg -- that are making contributions that advance Iceberg's functionality.

"Apache Iceberg has emerged as the industry standard for managing historical data at scale, with adoption happening with all of the big vendors now and becoming the standard over Delta Lake," Catanzano said. "Its open table format … addresses critical needs in large-scale data management."

Because Iceberg enables users to combine disparate data types to fully inform applications, can connect to various systems and is capable of handling petabyte-scale workloads, it has become a popular means of helping to train generative and agentic AI models and applications.

Among the data management vendors now providing support for Iceberg are AWS, Cloudera, Databricks -- even though it was one of the founders of Delta Lake -- Dremio, Google Cloud, Oracle, Qlik and Snowflake.

Now StarTree is planning to do the same by adding native support in StarTree Cloud with a combination of customer feedback and market observations providing the impetus, according to Kishore Gopalakrishna, StarTree's co-founder and CEO.

"On the customer side, we saw a clear trend of organizations fully committing to Apache Iceberg as the foundation of their open lakehouse, and they didn't want to move their data into proprietary formats just to make it usable," he said. "At the same time, we've been tracking the rapid rise of Iceberg as the de facto open table format for managing large-scale analytics data in cloud object stores."

Key capabilities of StarTree's support for Iceberg include the following:

• Direct querying of Iceberg tables from StarTree, eliminating costly and complex data egress.
• Real-time indexing and aggregation of data, including unstructured data formats such as text.
• Local caching so users can store frequently used data in the same memory space as applications to reduce the latency and improve the concurrency of queries.
• Intelligent prefetching that proactively loads relevant data to speed performance.

While each has value, eliminating the need to move data is perhaps the highlight of StarTree's support for Iceberg, according to Catanzano.

"It fundamentally changes how companies can utilize their lakehouse investments for real-time applications," he said.

In addition, real-time indexing and aggregation is a valuable feature, Catanzano continued.

"It enables the high-performance, low-latency access that makes StarTree uniquely positioned to deliver interactive insights directly from the data lakehouse," he said.

Next steps

Beyond moving its native integration with Iceberg from the preview stage to general availability, StarTree's roadmap includes providing more capabilities that enable customers to build agents and improving its observability capabilities, according to Gopalakrishna.

To better aid agentic AI development, in addition to adding  support for MCP, the vendor has plans to integrate with vector embedding models. Meanwhile, tools such as anomaly detection tool StarTree ThirdEye provide observability.

"Our focus is on helping companies build intelligent, responsive and scalable data products that meet the needs of both human users and AI agents," Gopalakrishna said.

Improving ThirdEye, possibly by adding new machine learning capabilities, is wise, according to Catanzano.

In addition, he suggested that StarTree integrate with AI vendors to provide users with tools to develop sophisticated AI applications trained on now easily accessible data from their data lakehouses and develop tools to aid customers in specific verticals.

"They may [create] industry-specific solutions that address unique real-time analytics challenges in sectors like e-commerce, financial services and IoT where immediate insights drive business value," he said.

Eric Avidon is a senior news writer for Informa TechTarget and a journalist with more than 25 years of experience. He covers analytics and data management.