your123 - stock.adobe.com
After a year of iteration and development, the DataStax Astra Streaming service is generally available as of Wednesday.
The Kesque technology became the original foundation for the Astra Streaming service, which provides a commercially supported event data streaming capability. While Astra Streaming started out only supporting Pulsar, with the new release DataStax has extended the technology to support Apache Kafka, open source RabbitMQ and Java Message Service (JMS) data streams as well.
While Astra Streaming now has extended its protocol support beyond Pulsar, it still has a few gaps to fill. Among the other popular formats that Astra Streaming doesn't currently support are MQTT and Apache ActiveMQ, which are commonly used for IoT applications.
Among DataStax's Astra Streaming users is Michael Smith, director of engineering at financial services company Commonstock in San Francisco.
Michael SmithDirector of engineering, Commonstock
The company's data architecture provides services for financial market data, user portfolio data and user comments, and the services communicate with each other via APIs.
Also, Commonstock has a service that recommends content to display in a user's feed. In order to make recommendations, the platform needs to get metadata from multiple services in real time. That's where Astra Streaming fits into the company's IT architecture.
"We use Astra Streaming as a mechanism to generate events on services that other services can subscribe to," Smith said.
An example is when Commonstock recommends content based on whether someone a customer is following clicks the "like" button for a trade that someone else made, Smith said.
Event data streaming beyond Pulsar with Starlight
The effort to enable the Pulsar-based Astra Streaming service to support other event data streaming technologies comes from a set of open source efforts led by DataStax known as Starlight.
Starlight provides a compatibility layer for different protocols, on top of Pulsar. For Kafka, RabbitMQ and JMS, DataStax developers working on Starlight had to understand how each specific protocol works to support users that want to run those protocols with an Astra Streaming deployment, according to Chris Latimer, vice president of product management at DataStax.
The support for Kafka, RabbitMQ and JMS in Astra Streaming is bidirectional, meaning the service can both import and export event data streams in any of those formats, in addition to Pulsar.
Connecting Astra Streaming to a larger data platform with Astra DB
Streaming data on its own can help organizations with operational dashboards and messaging. Streaming data can also be useful for data analytics and machine learning, but it often needs to be in a database, which is where DataStax's Astra DB cloud service fits in.
In March, DataStax said it was using the Astra Streaming technology to enable change data capture capabilities for Astra DB. While Astra Streaming was not generally available at that time, DataStax was committed to providing full support, but only for the change data capture application with Astra DB, according to Latimer.
Now that Astra Streaming is generally available, Latimer said DataStax is supporting Astra Streaming with Astra DB for any application. For example, data can now be ingested into Astra DB and then used for data analytics.