Dynatrace overhauls AIOps back end, beefs up log analytics
The AIOps vendor unveiled a new approach to data management that has enterprise customers such as BT hopeful it could eliminate the need for separate log analytics products.
Dynatrace is set to replace the data management underpinnings of its cloud-based AIOps tools, beginning with new support for log analytics that could set the stage for fresh competition with Splunk, Elastic, Sumo Logic and others.
The new architecture, dubbed Grail, culminates more than three years of engineering work behind the scenes, Dynatrace officials said. Grail includes a massively parallel processing engine, a new query language and a data lakehouse, so named because it blends elements of both a data lake for speedy ingestion of unstructured data and a data warehouse for quickly processing structured data. This type of hybrid architecture is also marketed for general-purpose data analytics used by vendors such as Snowflake and Databricks.
In the case of Dynatrace, Grail is built specifically to hold data that feeds the company's AIOps products for observability. That includes log analytics data that can now be stored directly alongside the events and trace information Dynatrace already handles based on its roots in application performance management.
"Grail is a massively parallel processing engine for managing data," said Rick McConnell, CEO at Dynatrace. "The first use of it, we will apply toward log management and analytics, because our perspective is that the log market with Splunk, Elastic and others has, to date, still been largely siloed and very isolated."
Dynatrace log analytics: Third time's the charm?
This is not Dynatrace's first attempt at incorporating log analytics into its AIOps tools, noted Gregg Siegfried, an analyst at Gartner. In February 2021, it added log data support to its software intelligence platform. In September that year, it acquired SpectX to add further log analytics features. But by February 2022, some users expressed frustration after initial attempts to link Dynatrace's log monitoring tools with back-end business intelligence (BI) systems.
"This will be bigger than just logs, but that has been kind of a gap for them," Siegfried said.
BT Digital, a division of the London-based multi-national telecom, signed on to replace dozens of separate IT monitoring tools and ticketing systems with a combination of ServiceNow and Dynatrace products in June. The company is about 25% of the way through this conversion project, which will encompass some 80,000 virtual hosts in all. BT hasn't yet tested Grail -- but one of the telecom's IT leads is hopeful Grail will further reduce the number of separate IT management tools his company uses.
"Assuming it proves itself in a proof of concept, what it will mean for us is that we've got a single place to go to for log analytics and root cause analysis," said Alex Bell, ServiceNow delivery and engineering lead at BT. "We think there are three or four other [log management and analytics tools] that we can just now bring under that single capability."
BT plans to replace multiple IT monitoring tools with Dynatrace to tackle all stages of AIOps.
Overall, BT has an ambitious set of goals for its AIOps work with Dynatrace, up to and including what it calls ZeroOps, also known in the industry as NoOps, in which systems become automatically self-healing without human intervention.
For now, BT is starting with AIOps automation for relatively simple cases, such as server restarts. But being able to store more data for analysis by Dynatrace could help the effectiveness of automated root cause analysis and event correlation under AIOps in the future, Bell said.
"The more confident you are in that root cause, the more quickly you can get to the root cause, and the more automatically you can get there, the more effective that ZeroOps vision will be for us," he said.
More data, more competition
Dynatrace will introduce a new proprietary query language, Dynatrace Query Language (DQL), when Grail becomes available to SaaS customers next month. The vendor's execs claim this will improve SaaS performance, economics, and integrations with third-party data analytics systems, including third-party business analytics and BI tools.
That query language is based on a graph API similar to the open source GraphQL API that's begun to gain momentum in advanced data management circles. VMware is taking its own approach to tools that integrate a knowledge graph with its Aria portfolio launched in late August; a cloud-native SBOM tool in the works from CNCF will incorporate a graph database.
Assuming [Grail] proves itself in a proof of concept, what it will mean for us is that we've got a single place to go to for log analytics and root cause analysis. We think there are three or four other [log management and analytics tools] that we can just now bring under that single capability.
Alex BellServiceNow delivery and engineering lead, BT Digital
Knowledge graphs and graph APIs are gaining popularity among IT automation and management tools because of the increasing complexity of relationships between microservices apps, which knowledge graphs can make inferences about more quickly than relational and NoSQL databases queried via REST APIs.
Dynatrace already has a knowledge graph feature it calls Smartscape, but Grail and DQL will allow for more custom, high-performance queries of multiple types of data within the same storage repository, instead of having to bridge between separate pools of data as Smartscape does currently, said Steve Tack, senior vice president of product management at Dynatrace.
"[Customers] will be able to bring their own business logic, their own queries, in a much more freeform way [for] use cases [such as] fraud detection that we don't provide out of the box with Dynatrace," Tack said. "But now we're giving them access to data from logs, traces, user experience sessions and more, and they're going to be able to create their own use cases."
Pricing has not yet been set for Grail-based services. But applying Grail's massively parallel data processing engine will also mean it can handle data ingestion for one-tenth the cost of competitors, McConnell said. Dynatrace also plans to price Grail based on the number of queries rather than the amount of data stored or users with access, he said.
While it's clear Dynatrace has its sights set on competitors such as Splunk, Elastic and Sumo Logic, it will need to demonstrate the effectiveness of Grail before it can unseat established log analytics competitors, Siegfried said.
"Splunk Enterprise effectively created the log analytics market as we know it today, and they have a next-generation [observability] product in SignalFx. But they haven't yet proven they know how to sell application performance monitoring," he said.
"At the same time, application logs are a fraction of what Splunk deals with -- they also deal with logs for security and IT operations that aren't necessarily connected with a single application -- and it's not clear whether the Grail aspect of Dynatrace will be positioned as that kind of a general-purpose log analysis tool."
Both Splunk and Dynatrace focus on large top-down enterprise sales for observability tools, while rival Datadog has also begun to encroach on their turf using a bottom-up, developer-focused strategy, landing its own major deals with the likes of Mendix and showing strong growth in recent earnings reports.
"Datadog is firing on all cylinders, both on the product development side and the sales and marketing execution side," Siegfried said. "Dynatrace tends to focus on larger companies and larger deals, and Datadog has generally done the land-and-expand thing on a team basis. But they're also starting to get noticed at the executive level as well."
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
Dig Deeper on IT systems management and monitoring
