Embattled New Relic cuts Kubernetes observability overhead
New Relic made its Kubernetes observability agent more efficient and added vulnerability management to its APM tool but is struggling financially as open source tools proliferate.
New Relic improved Kubernetes observability performance, amid other updates to its New Relic platform, but financial reports this week pointed to struggles for the vendor amid internal upheavals and mounting competition.
The news came on the heels of a year of upheaval for the company, following a drastic overhaul of its observability products and the appointment of a new CEO, plus the ongoing migration of its back-end infrastructure to AWS.
In a report issued May 13, New Relic beat its own revenue guidance for the fiscal year ending in March, and its sales increased in its fiscal fourth quarter by 19% to $205.8 million, beating Wall Street expectations. But the company also operated at a net loss of $229 million for the year, compared with a net loss of $171 million for fiscal 2021, and its annual financial report showed that the 15,400 active customers it reported in the first fiscal quarter of 2021 had fallen to 14,800 by the fourth quarter of fiscal 2022.
New Relic is facing the same fundamental Innovator's Dilemma as most of its commercial observability competitors, including Splunk, which has also undergone executive turnover and reported cash flow shortfalls during the last year.
In part, these vendors are scrambling to maintain their customer bases and keep up with ever-more-complex cloud-native technologies such as Kubernetes and service mesh. Open source projects also offer an unprecedented number of free alternatives, from OpenTelemetry for distributed tracing, Prometheus for Kubernetes metrics and Grafana dashboards, to a multitude of open source logging tools such as the Elastic Stack and Amazon OpenSearch.
"All the original [application performance monitoring (APM)] vendors are going through a transition now," said Andy Thurai, vice president and principal analyst at Constellation Research. "Some are more mature than others, but it is a growing pain that they are all going through -- I don't see any major player in APM and infrastructure monitoring knocking it out of the park in the near future."
Still, some have fared better than others. Datadog, for example, reported 83% revenue growth for the first quarter of 2022, following 84% revenue growth for the fourth quarter of 2021 and overall revenues of $326 million for 2021. Datadog also reported that it gained 100 more customers with annual contracts valued at $1 million or more in 2021 compared with 2020.
New Relic boosts Kubernetes observability scale for large customers
One bright spot for New Relic in its earnings report this month is an increase in customer accounts worth more than $100,000, even as its overall customer count fell. Such customers numbered 862 in the fiscal first quarter of 2021, compared with 1,099 in the fiscal fourth quarter of 2022, according to the company's annual financial report. New Relic officials also said such large-scale customers are looking to manage fewer, larger Kubernetes clusters as production container deployments grow, which this week's product updates reflect.
"When you are a large streaming company or telco and you're moving your workloads from VM clusters to Kubernetes clusters, for [network] efficiency, you can't have that many small clusters," said Ishan Mukherjee, group vice president of product at New Relic. "One thing that was prohibiting them from using APM and observability tools in general is the memory footprint and the overhead of the agents, and we have significantly reduced that so that [these customers] can use it at 1,000-node scale."
The New Relic platform now offers two methods of data collection within Kubernetes clusters: via Extended Berkeley Packet Filter (eBPF) instrumented with New Relic's Pixie open source tool, or through a more traditional software agent. The eBPF approach has gained favor in large-scale shops for observability and security in recent years because of its lower resource overhead, and will be New Relic's focus in the future.
However, many New Relic customers also still use its agents, which it has made more efficient with this week's release, Mukherjee said.
"We're reducing the memory utilization [on Kubernetes nodes] and using a more advanced [data] sampling technique," Mukherjee said. "With Kubernetes clusters specifically, the volume of inter-entity transaction chatter is 10 to 100 times higher than in VM clusters, so if you use the same [data] capture and sampling approaches, you will end up having [exponentially] more data collected."
With this new release, the New Relic agent samples data at a lower rate and excludes low-value chatter data to mitigate this problem, Mukherjee said. To set the stage for its shift to eBPF, this week's release also deepens the integration between Pixie, which New Relic acquired in late 2020 and donated to open source last year, and the New Relic platform. Last year, the platform began to support real-time views of eBPF data in running clusters in its topology dashboard and will now ingest historical data for comparisons beyond the real-time window of six hours.
Pixie isn't without its own memory overhead, roughly 512 MB per node. Customers can reduce data retention time and use data compression features to keep memory usage manageable, according to New Relic officials.
Security add-on will increase pricing pressure
New Relic also rolled out its first security monitoring feature this week, adding vulnerability data to its APM dashboard within the New Relic platform. New Relic is the latest among many APM and observability vendors to diversify into the security realm -- Splunk, Sysdig, Sumo Logic, Elastic, Dynatrace, AppDynamics and Datadog, among others, have already connected security and observability data within their product lines as DevSecOps becomes a priority for platform engineers and SREs.
At least initially, New Relic isn't looking to create its own general-purpose security tools, Mukherjee said.
Ishan MukherjeeGroup vice president of product, New Relic
"Our user base is primarily developers, and we're going to be developer-focused, harvesting vulnerability data from our instrumentation and working with the Laceworks and Snyks of the world," he said. "We'll focus on the 'left' in 'shift left' and send security signals to developers in preproduction environments."
New Relic Vulnerability Management is in beta, and the vendor is piloting other security-related feature ideas such as security information and event management (SIEM) tools; database posture management and bespoke integrations with specific applications such as SAP, Office 365 and popular databases.
For now, Vulnerability Management is free to customers of Data Plus, a pricing tier with a 90-day retention period launched April 14 and priced at $0.50 per gigabyte. Data Plus combines features such as high-performance querying and FedRAMP and HIPAA compliance support that were previously priced separately. The vendor estimates the combined Data Plus is 56% cheaper than buying separate features, butthere aren't specific plans to adjust data ingestion pricing to for lower-access tiers of archival observability data similar to those made by competitors such as Sumo Logic and Splunk, according to Mukherjee.
Thurai predicted New Relic will need to make further pricing concessions as observability data volumes grow, a tough decision while it faces financial worries. Without it, the company risks losing further customers to free and open source tools.
"Pricing seems to be a sore pain point for many large digital enterprises," Thurai said. "Given the choice and variety that is available in the market ... if New Relic doesn't reduce pricing fast enough, they will either lose customers to smaller players or to open source."
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.