A former LinkedIn architect now at the helm of a DataOps startup has his sights set on a new approach to data storage for observability in the age of AI.
StarTree was founded in 2019 by the LinkedIn engineers who developed the open source Apache Pinot distributed data store. One of those founders was Kishore Gopalakrishna, now CEO at StarTree, which until recently had focused on offering an online analytical processing (OLAP) platform for business intelligence. However, as observability became infused with growing volumes of AI data, prompting IT ops teams to draw on DataOps practices, Gopalakrishna spied a new opportunity.
"If you look at metrics, the way time series [data] was stored was very different early on, [but] the concept of labels and dimensions was ... introduced, and that made the data structure and the data layout more similar to OLAP stores," Gopalakrishna said during an interview on Informa TechTarget's IT Ops Query podcast. "That's why something like Pinot [is] becoming a lot more relevant for observability, because people are adding a lot more richer context [to their data]."
Pinot [is] becoming a lot more relevant for observability, because people are adding a lot more richer context [to their data].
Kishore GopalakrishnaCo-founder and CEO, StarTree
Apache Pinot and other OLAP systems categorize data according to multiple dimensions, such as time, date, location and activity type, which can then be used to query data at high speed and return results with low latency, even without high-end storage hardware. StarTree has begun to redevelop its platform to support a blend of metrics, logs and traces in the same distributed back end for observability, similar to the design approach other vendors have described as "Observability 2.0."
Gopalakrishna acknowledged that his company faces established competitors in the observability space that also take a distributed storage approach, such as Honeycomb and Dynatrace. He said StarTree will differentiate itself with its real-world experience building high-scale, high-performance systems for big businesses such as DoorDash, Razorpay and Stripe.
"If you look at the origin of Pinot, we focus on ... millisecond response latency and lots of concurrency," he said. "It's just an extension of what we've done for observability use cases."
StarTree also allows large enterprise customers to host its service on their own back-end public cloud infrastructure, in contrast to many observability competitors, Gopalakrishna added.
Kishore Gopalakrishna
He predicted that, in the future, AI models will fundamentally change DataOps as well, both inside and outside the observability market. Large language models are capable of attaching complex dimensional metadata, or embeddings, to streams of events that support later analysis without requiring the data to be converted into a structured format. That opens up new possibilities for natural language queries, Gopalakrishna said.
"[That could] lower the bar in terms of how much knowledge you need to have before you are actually leveraging logs and metrics for root cause analysis or debugging a subsystem," he said. "There is still a long way to go to get [there] from where we are. ... The challenge there right now is the cost of these models is still high. It's not as low as we would like for running this on an event-by-event basis."
Beth Pariseau, a senior news writer for Informa TechTarget, is an award-winning veteran of IT journalism covering DevOps. Have a tip? Email her or reach out @PariseauTT.
Kishore Gopalakrishna
