Confluent Platform 6 brings cluster linking to Apache Kafka

Event streaming vendor Confluent adds new cluster linking capabilities in its latest platform update, bringing more scalability for deployments that span multiple clouds.

Confluent released Confluent Platform 6, an update that integrates new capabilities on top of the open source Apache Kafka event streaming platform.

The stream processing vendor, based in Mountain View, Calif., revealed the update Aug. 24 during its Kafka Summit virtual conference, which ran Aug. 24-25, but did not say when it will be generally available.

Among the enhanced capabilities in the latest version of the platform is the ksqlDB event streaming database, which was first introduced in November 2019.

Confluent also integrated a series of scalability improvements it has referred to collectively as Project Metamorphosis. As part of its scalability capabilities, Confluent put a cluster linking feature in the update that aims to make it easier for users to connect different Kafka event streaming data instances.

Confluent Platform 6 metamorphosis

Confluent Platform 6.0 is the first fruit of Project Metamorphosis, and it is clear that more capabilities are coming, said IDC analyst Stewart Bond.

"Confluent Platform 6.0 is adding features around Kafka that will make it easier for developers to build cloud-native data applications against and increase adoption of real-time stream processing workloads," Bond said. "Business has always been in real time, but technology limitations of the past have constrained it from computing in real time."

Kafka cluster linking

Specifically regarding the new cluster linking capability, Bond noted that at a high level, cluster linking may look like basic federation, but underneath is where it is more than that.

Confluent Platform 6.0 is adding features around Kafka that will make it easier for developers to build cloud-native data applications against and increase adoption of real-time stream processing workloads.
Stewart BondAnalyst, IDC

Bond noted that cluster linking preserves offsets across the cluster. In an independent cluster, offsets are used to manage the sequence in which data messages are delivered to a consumer and ensure consumers don't get multiple copies of the same message.

"Offsets are very important in horizontally scaled environments where multiple instances of processes can be running at the same time, reading from the same cluster," Bond said. "With cluster linking, multiple clusters can be federated, data can be shared across clusters, and with offsets being preserved, processes can also be shared across clusters. "

Cluster linking uses the Kafka binary protocol. Addison Huddy, group product manager at Confluent, noted that with cluster linking, Kafka clusters running across different deployments across cloud providers or on premises can be connected together.

"Cluster linking allows software architects to create a global mesh of Kafka," Huddy said.

ksqlDB now production-ready

The ksqlDB technology is a database that is purpose-built for Kafka event streaming. Confluent Platform 6.0 promotes two of ksqlDB's most powerful features, its query layer and connector management, to general availability, said Michael Drogalis, Principal Product Manager at Confluent. The platform update itself is not yet generally available.

Drogalis explained that the query layer provides a low-latency interface for asking point-in-time questions. The connector management capability enables users to source and synchronize data with Confluent's portfolio of more than 100 data source connectors.

Cluster linking in Confluent Platform 6
Confluent Platform 6 integrates a cluster linking capability that enables users to create a mesh of multiple Kafka instances running on-premises or in the cloud.

"Both of these features have been hardened over the last year through continuous usage, both from our customers and our own cloud operations," Drogalis said. "KsqlDB has seen adoption for building both materialized views and streaming ETL [extract, transform, load] pipelines, both of which are crucial for building modern applications."

Next Steps

Confluent goes IPO as Kafka event streaming goes mainstream

Dig Deeper on Database management

Business Analytics
Content Management