Grafana Labs targets the rising price of AI observability

Grafana speeds up search, shrinks archive data

In addition to the general availability of agent monitoring features in Grafana's Cloud AI Observability dashboards, this week's updates to Grafana's Cloud Logs, Adaptive Metrics, and Adaptive Traces offered users more control over the explosion of AI telemetry data. These included a new indexing approach for Grafana's Loki-based Logs service, based on its April acquisition of search-and-query vendor Logline.

The new approach stores high-cardinality values, such as user IDs, transaction IDs, or IP addresses, in an object storage system rather than a traditional file system. This means Loki can quickly find specific identifiers without scanning the entire data set. Early benchmarks for Logline's IP integrated with Grafana Cloud Logs showed a query that had previously scanned 3.5 TB reduced to 8 GB—a 99.7% reduction in data scanned.

Torsten Volk

Helping customers find the needle in the haystack has become critical, as AI has created an escalating number of exponentially larger haystacks that have become too expensive for many enterprises to search, according to Torsten Volk, an analyst at Omdia, a division of Informa TechTarget.

"Anything that helps reduce the size of these haystacks through intelligent sampling based on an organization's real-world monitoring and root cause analysis requirements is required," Volk said. "This represents a clear step forward in making production AI accessible to the masses."

Further Grafana updates this week included fine-tuning the filters it uses to reduce long-term observability data storage volumes:

• Adaptive Metrics Auto-Apply is generally available. This feature automatically applies the system's usage-based recommendations for data segments so teams no longer have to review every analysis. Users choose Coverage-first (applies every recommendation so the segment keeps up with new usage) or Cost-first (limits estimated series growth and applies increases only up to the budget from reductions in the same cycle). Exemptions can be created in advance so that critical metrics are not aggregated.
• Query-by-ID for dropped traces. Trace data dropped by Adaptive Traces remains queryable by trace ID for up to 24 hours after ingest. This is useful when a log line, an exemplar, or another system provides a specific trace ID for investigation. The first lookup of a dropped trace within that window writes it to Grafana's Tempo distributed tracing backend for long-term storage; traces that aren't looked up within that window are permanently discarded. Nothing needs to be enabled or configured—the behavior applies automatically to every Adaptive Traces tenant.
• Adaptive Logs Archive entered public preview. The service lets teams route low-value logs to their object storage bucket instead of indexing them in Grafana Cloud, preserving the data at a lower cost for later audit, compliance, or investigation while still delivering the savings from not storing unused logs on the platform. New features that reduce the amount of data stored in the archive include pattern-based drop recommendations, drop rules, exemptions, and temporary pauses.

AI observability concerns weigh on IT pros

The 2026 State of Observability Report released this week by Grafana Labs and ETR, a Futurum company, reflected observability users' concerns about teams using agents to operate their systems, and understanding the behavior of the agents themselves.

As AI moves into production, 68% of the 300 IT and engineering leaders surveyed have moved AI for observability past investigation into pilots or production. But confidence depends on the task and readiness varies: 85% report comfort with read-only investigation, compared to 52% for full autonomous remediation.

Teams also need to watch the agents they ship. Some 80% of surveyed organizations are building or operating LLM-powered features, yet 45% have pieced together AI observability from open source while only 25% run a purpose-built product.

Grafana Labs' product updates this week covered both sides of this equation: AI for observability, so agents can investigate and act with people in control, and observability for AI, so teams can see and trust the agents they run.

AI is not going to magically know your services; you have to let it build that memory, and then you still review what it surfaces.

However, organizations are still only cautiously relying on agents for resource optimization and troubleshooting while humans remain in the loop.

"The [AI] assistant still needs a human as the glue," said Oren Lion, director of software engineering at TeleTracking, makers of a healthcare operations platform in Pittsburgh, Pa. "The assistant can run the investigation and draft notes, but someone has to decide if the result is the real root cause or just what might have been a plausible story."

It all takes time as well.

"AI is not going to magically know your services; you have to let it build that memory, and then you still review what it surfaces, which is the baby step toward more autonomy rather than a full hand-off," Lion said. "The assistant is useful when it can stay inside the tool and show you the query it ran. You still have to trust but verify, and the cost control only works if the adaptive features are actually dropping the right things without hiding the signal you need later."

The observability team at SailPoint, an identity protection security provider, quickly migrated to Grafana Cloud Traces with Grafana Assistant, which it used to write sampling rules, migrate dashboards, and write and interpret queries in Grafana's TraceQL language for traces.

Such a migration, along with the sampling-rule work, could previously take a month or two, but the SailPoint team completed the migration in one week with one engineer, said Omar Lopez, a DevOps manager who leads the observability team at SailPoint, during an ObservabilityCon keynote presentation.

"Grafana Assistant has been a tremendous help," Lopez said. "And we're not only moving services; we're also onboarding new teams."

With Grafana Assistant, new teams can quickly learn and write PromQL, a time-series query language, and create dashboards, Lopez said.

"When they started using it to run investigations, this really helped to reduce incident response time or incident investigations from hours down to minutes," Lopez said.

"A few weeks ago, our internal team had an incident. Luckily, it was in dev, and no external customers were affected," he said. "But I wanted to test out investigations myself, so I jumped into the incident Slack channel, and I had Grafana run an investigation. Sure enough, a few minutes later it came back. It knew exactly what happened, and it even offered a couple of solutions."

Bruce Gain is founder and principal analyst at ReveCom. His byline has appeared in Wired, PC World, CIO, Technology Review, Popular Science, and EETimes.