Observability maven 'cranky' about AIOps embraces GPT

Honeycomb, founded by one of the early pioneers of observability, previously shunned adding AI features for its products, but has jumped on board with natural language queries driven by generative AI.

Charity Majors, CTO and co-founder at Honeycomb, is widely credited for coining the term observability to denote the holistic understanding of complex distributed systems through custom queries. She describes herself as "salty" in general about AIOps and machine learning (ML) features in IT ops tools. But this week, Honeycomb revealed that it will inject AI into its products for the first time via OpenAI's GPT API.

This new partnership is the basis for an experimental Honeycomb feature called Query Assistant, which supports natural language queries on observability data and is now available free to all customers. The feature can be turned off by teams wary about data privacy, and no user data is passively sent to OpenAI, according to the company's press release.

TechTarget Editorial caught up with Majors this week to find out why the value of AI in the case of OpenAI's generative AI differs, in her view, from that of AIOps.

Does Honeycomb have its own query language? How did people do queries before this?

Charity Majors

Charity Majors: It was a graphical query builder, and honestly, it's kind of the slow, manual way to do things. We also have this thing called BubbleUp, which is not AI or ML, but it's more effective at finding problems. If you're sending data in with some intention, then you, the human, can draw a bubble around something on the graph that you're concerned about: 'I care about this.' In the background, we compare the inside of the bubble with what's outside of the bubble, and then we sort and diff them, so all the things that are different about what you care about are right there on top.

That little loop is what 99.9% of all system debugging boils down to: 'Here's the thing I care about, it has meaning. What is different about it?' Then the machine immediately tells you, 'Oh, it's different because all these errors are for devices coming from Android, that were on this version, this build ID, that you're using this app language pack for, in this region,' and then you usually immediately know what the bug is. If you can't find those things, then it might take you days or weeks to figure out what's going on.

What do natural language queries add to Honeycomb for customers?

Majors: It's kind of a great equalizer. Query Builder is great if you already understand SQL well and you already know your data fairly well. But [Query Assistant] allows less sophisticated developers and people who aren't even engineers to go, 'We just deployed something. What's slow about it, or what changed?' Or 'Show me the slowest endpoints.' Or 'Which users are experiencing the most bugs or the most errors?' Or 'I just changed something about the payments endpoint. Is anyone experiencing queries over 5 seconds?' The tool completely gets out of the way. This is not great for expert-level stuff, but it's amazing for getting started. And honestly, getting started is most of what you do over and over again in a complex system.

I am on record as being incredibly cranky about most of the AI offerings in the market. But I'm really excited about this.

Honeycomb's always been, inevitably, kind of a complex tool. You can ask any question to understand your system, and you're not locked into the custom metrics or anything. But our philosophy of developer tools is that they should try and get out of your way. You should only be trying to focus on the question you're trying to ask, the problem you're trying to solve -- not also trying to figure out the tool that you're trying to solve it with as an intermediary. This is such a beautiful step forward. Anyone can tweak a query.

It also dovetails with our whole philosophy -- we're not about tossing AI on the salad and just being like, 'Woo-hoo!' I am on record as being incredibly cranky about most of the AI offerings in the market. But I'm really excited about this. It's going to help engineers be better at their jobs in a meaningful way.

What makes Query Assistant different from 'tossing AI on the salad'?

Majors: My big gripe with AIOps is that it usually is a solution you shouldn't need for problems that shouldn't exist. One of the things that [AIOps vendors] often tout is, 'If you're getting flooded with emails from alerts, then we can tell you which ones are important.' But you shouldn't have to send millions of alerts. They're not useful. What they're basically saying is, 'OK, humans can't make sense of all this noise anymore. So, let the machine do it for you.' But that's worse, because then if the machine alerts you in the middle of the night, and you can't figure out why, now you're twice as frustrated.

This is where Bayesian math comes in: Even if the thing happens one in a million times, and if the thing happens, it has a 90% success rate, something like 80% of all alerts are still going to be false positives. The most expensive thing that you can do in reliability engineering is false positives. You're just training people not to pay attention to the alerts -- it wears people out. And then when something does go wrong, the whole point of this whole [tech] industry is understanding your systems. It's not outsourcing understanding. Machines can find signals, machines can crunch numbers and tell you if there's a spike or a blip, but they can never tell you if that was good or not. They can't tell you if that was intentional or not. Only humans can attach meaning to things, and what most AIOps is doing is trying to take the meaning-making away from humans and give it to machines. And that's just busted, because eventually a human's going to have to come along and understand it at the end of the day, and you're hobbling them and crippling them.

What else might Honeycomb do with natural language queries in the future?

Majors: We don't have a way to preserve state [in queries]. This is the one downside -- you can't iterate on your queries. It's a fresh slate every time. It would be really cool if we could let people iterate on and refine their queries, and even do learnings across accounts so that we can suggest for people who are just getting started with a MySQL data set, 'Here are some really interesting things that people using systems like yours do.'

There's also some really interesting stuff [around shared knowledge] that I can't wait to get into. If I'm working on my corner of the system, I know it intimately, but when you're debugging, you have to debug the entire system. And most of the knowledge of that system is not in my brain, but little pieces of it are in your brain, my brain, Caroline's brain, Ben's brain. Being able to access that, if the AI was a little angel or devil sitting on your shoulder suggesting things, telling you, 'Here's what the other people in your system are doing to interact with it' -- this is how we learn to become experts, by watching other experts. If that information can be served up to you on a platter, it can make you so much smarter and more effective at your job because you're drawing on the wisdom of the crowd.

Editor's note: This Q&A has been edited for clarity and conciseness.

Beth Pariseau, senior news writer at TechTarget Editorial, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.