Dynatrace execs dish details on Bluebox autonomous AI

Closing the loop between production observability and the early stages of the software development lifecycle is a longstanding ambition in DevOps, but AI agents are prompting one vendor to start over from scratch.

Rolled out to early-adopter customers this month, Bluebox.ai is a new venture from Dynatrace that entirely removes humans from the process. Instead, Dynatrace observability and SRE agents continuously feed data, prompts, skills and other instructions to coding agents throughout the software development process and in response to production issues. Those coding agents then make changes to the software and redeploy it as needed.

Two executives from Dynatrace, founder and CTO Bernd Greifeneder and chief technology strategist Alois Reitbauer, spoke with TechTarget about the project and how they believe enterprise software development will change in the age of AI agents -- including what comes next for humans.

Editor's note: The following was edited for length and clarity. Greifeneder and Reitbauer were interviewed separately.

What's new and what's ahead for Dynatrace in agentic observability?

Bernd Greifeneder: What we see with our customers is that they have become bimodal. On the one hand, they drive all the AI investments to make their existing deployments and teams more productive. But in parallel, more and more start initiatives through innovation teams to build a pure agent-led setup [and] to figure out, aside from their core business, what the new way of creating and delivering software looks like.

Our product release for [this week] is about helping existing initiatives with a higher grade of automation in their software delivery and resilience through our next-generation investigation agents. …These customers are in that first mode of the two -- human-led -- where you use AI to assist the humans in regular processes. The other mode is agent-led, where 80% of observability use in Dynatrace is handled by AI agents.

TechTarget: What exactly is new with the release for mode one? Because Dynatrace already has root cause analysis and an SRE agent.

Greifeneder: It's the whole depth and breadth to investigate not only within a single identified root cause, but also across multiple causes, making it much easier for the team to handle concurrent items. It's also hyperscaler infrastructure integration, along with partners like ServiceNow, Atlassian and PagerDuty.

So that's aimed at that human-led 'mode one' audience. But I'm sure Dynatrace has plans for the second mode.

Greifeneder: More than plans. We have a sub-brand for Dynatrace that caters particularly to the second mode: Bluebox.ai. What that does is take the human completely out of the loop of the entire SRE and DevOps cycle. It's telling coding agents how to instrument their applications based on the production architecture and how to improve the instrumentation using OpenTelemetry.

Where coding agents are limited is in gaining a true awareness of what's going on in the production environment.

Then tests happen, then rollout to production, and Bluebox continuously assesses what's going on and advises the coding agents with concrete suggestions on what to change, even before new features are done. Then there is the rollout, and naturally, when issues happen, the full investigation happens autonomously and goes further than in the first mode, because all the investigation, analysis and remediation steps go hand in hand with the coding agents automatically.

Usually, customers set up a new set of coding agents, and then they get stuck because those coding swarms are really great at reading specifications, processing knowledge documents, and spitting out tons of code. They're even good at reading short, local snippets, logs and so forth. But where coding agents are limited is in gaining a true awareness of what's going on in the production environment.

Just to connect some dots, last year, Alois, you were on my podcast discussing the difficulty of getting observability data from production back into code, particularly the challenge of maintaining a global understanding of the code and what changing a function would mean in production. You also talked about training small language models to help with that. Is that what's been going on behind the scenes with Bluebox?

Alois Reitbauer

Alois Reitbauer: We handle some stuff with skills. I see skills becoming more and more that intermediate layer, where you're getting better at instructing the larger models. And to some extent, I think processes will also switch back to more deterministic code generation. We still see the deterministic work that we are doing playing a lot of that role, like the problem analytics, the graph analytics we do on top of Smartscape informing decisions. Because with large models, that's where prices get crazy.

Yes, anybody can build an SRE agent. You can link Claude to your observability platform and get reasonable output. It might create [pull requests]. What we are tracking against is what the actual output quality is, like how often the output is actually approved and used in the further process, and optimizing on time and resources. I think the future will be defined by whoever can do things both fast and cost-efficiently. If I can spend $25 on every single interaction, that is fine, but nobody has the luxury.

Right now, this doesn't come out of a small language model. I do still believe that this will be a differentiator in the future, but for now we built our own fine-tuned larger model. I think we might, at some point, take that very powerful model and then distill it into a smaller language model. But we start with the most capable first.

How do you keep the Bluebox agents honest? What kinds of safeguards are there in place to protect against this autonomous AI hallucinating or giving inaccurate information?

Greifeneder: This is why we always strive to not just throw everything into one LLM, but actually use deterministic algorithms as much as possible, and then use the agents to take a probabilistic approach to drive the next steps in the reasoning. But this is also why the whole idea of the AI lakehouse that we provide underneath Bluebox is so important, so that you turn petabytes of production data into kilobytes -- smaller chunks of highly accurate context -- and then you give that to the agents, so that you minimize all the risk of hallucination. Then, when new functionality is rolled out, you also capture results in test and staging before shipping it to production and establish an early feedback loop.

Additionally, in production, you use feature flags far more heavily than before to run additional A/B tests on these features and quickly turn them on and off. This is why Alois has been driving the OpenFeature standard, and why feature flagging is a key part of this entire Bluebox loop process.

How does a customer get from mode one to mode two, from AI-assisted to autonomous AI? What's the pathway from one to the other?

Greifeneder: To be honest, no one has done that yet. The core applications are too large and too critical for you to move all of this right away. The way to get there is therefore to think about which type of new application, as a business, you want to build in the second mode. And you intentionally pick a smaller one, one where you know you want to evolve rapidly, experiment rapidly, and start there because it will take a couple more months for this entire process to mature.

Once you have your first application working in that second mode, then you can onboard more and more. The lines will blur over time, but to be honest, you have to pretty much start from scratch.

What happens to humans when they go from in the loop to outside of it with something like Bluebox?

Reitbauer: Even with young engineers at the beginning of their careers, I'm having these conversations more than ever. And I just keep telling them: if you're trying to outrun a plane, you're going to lose. But if you know [destination] locations and the best way to fly there, you're not.

Building a good product is so much more than code.

There will still be a lot of work, but it will be very different work, and some people like it; others don't. If you define yourself as the best C+ coder on the planet, you might still have a reasonable career if you are, maybe, in your 40s. But overall, coding, to a great extent, is going away in favor of understanding system design, understanding experience, architecture and how you manage your agents to create something of value and taste.

I recently read an article that said there are more apps on the App Store than ever before, but nobody's using them. Building a good product is so much more than code. Building a good experience is so much more, especially since the pressure on product managers right now is almost unbearable in some cases.

In the past, as a product manager, you sat down, you designed a feature, you came up with an idea, and then you gave it to an engineering team. They came back like one sprint, two sprints later with the first idea, and then you kept going. What I'm seeing right now is they come back the next day, and then you work with more features in parallel, and honestly, at some point, you're running out of ideas. Also, I think you get exhausted at a certain point because it's so much brainwork. [You're] constantly being confronted with problems you have to solve.

My assumption is that people will work less, because I don't think that you can do deep thinking for 10 hours a day, and you don't need to engineer that much. I think you have a couple of deep hours of work in the morning. Then there's a lot of research, like lateral work. Maybe even taking some time off because your brain can't produce more output. I get that feeling from people that you need time to think through what actually feels good, how to build a good experience and what would really make a difference. I see this being infused in more and more roles.

Beth Pariseau, senior news writer for Informa TechTarget, is an award-winning veteran of IT journalism. Have a tip? Email her or connect on LinkedIn.