tiero - Fotolia

New Relic open sources Pixie observability platform

The Pixie Kubernetes-native observability platform is now available to the open source community after New Relic, which acquired the tech last December, contributed it to the CNCF.

New Relic has extended its commitment to open source software and the community by open sourcing its Pixie observability project.

In addition, New Relic has joined the Cloud Native Computing Foundation (CNCF) as a platinum member to help influence the direction of the cloud-native ecosystem, said Zain Asgar, general manager of Pixie and New Relic Open Source. Asgar was previously CEO and co-founder of Pixie Labs, which New Relic acquired in December. He joined the CNCF Governing Board and participated in a keynote address at KubeCon + CloudNativeCon Europe 2021 this week.

Pixie is a Kubernetes-native in-cluster observability platform that New Relic is in the process of contributing to CNCF as a new open source project under the Apache 2.0 license. Meanwhile, to extend availability of the project, New Relic will also expand its existing relationship with Amazon Web Services to provide the Pixie observability platform on AWS, Asgar said.

Aiming for comprehensive, low-friction observability

Pixie enables developers to see all of their applications' metrics, events, logs and traces with a single command-line interface command, said Ishan Mukherjee, co-founder of Pixie Labs and vice president at New Relic. It provides comprehensive observability for developers on its own, without the need for extra instrumentation, setting up dashboards or moving data off the cluster, he added.

As an open source project, developers can run an entirely self-hosted version of Pixie without third-party dependencies.

The assets New Relic contributed to open source include Vizier, the tool's data collection and query engine, which runs on a Kubernetes cluster where it collects data and stores it locally to execute queries for Pixie clients. It also contributed Pixie Cloud, Pixie Docs and the Pixie website.

Ishan MukherjeeIshan Mukherjee

Making Pixie ubiquitous

In addition, New Relic is dedicating the majority of the Pixie engineering team to continue its work on the open source project, said Bill Staples, president and chief product officer at New Relic in a blog post.

"It allows us to get Pixie in the hands of engineers across the world -- similar to Meteor for Kubernetes itself," Asgar said.

Asgar and Mukherjee envisioned building a SaaS product in the observability space, which was traditionally called application performance monitoring (APM), where vendors like New Relic, Splunk, AppDynamics, Datadog, as well as legacy vendors such as IBM and hyperscalers with CloudWatch (AWS) and Stackdriver (Google), play. However, as they realized traditional APM was merging with metrics, dashboarding and logs, they knew Pixie had to be all those things and more.

"Pixie Labs' technology provides observability/visibility/security for Kubernetes environments," said Stephen Elliot, an analyst at IDC. "It uses telemetry data collected entirely inside Kubernetes. Aimed at developers and IT operations teams that require Kubernetes visibility, it's an increasingly important capability and set of data for these teams. For the community, it adds more fuel to the fire and provides a robust growth opportunity for Kubernetes in general."

It allows us to get Pixie in the hands of engineers across the world -- similar to Meteor for Kubernetes itself.
Zain AsgarGeneral manager, Pixie and New Relic Open Source

The AWS connection

Meanwhile, in an AWS blog post by Colin Bookman, an independent software vendor senior solutions architect at AWS, and Mark Carter, general manager of observability services at AWS, wrote that they are "particularly enthusiastic" about the programmability of the Pixie platform. They also mentioned Pixie's use of Extended Berkeley Packet Filter (eBPF) to provide rich, automatic visibility of application events.

"Pixie makes observability easily accessible to developers," the AWS post said. "At AWS, we share that vision, to provide every developer access to high-quality observability data with minimal effort." As such, AWS is partnering with New Relic on the CNCF project and has appointed Jaana Dogan, an AWS principal engineer, to sit on the Pixie board.

Bookman and Carter cited Pixie's auto-instrumentation, programmatic data access and Kubernetes-native edge compute capabilities as key differentiators for the project.

"Pixie runs entirely inside Kubernetes as a distributed machine data system, meaning you don't need to transfer any data outside the cluster," the AWS post said. "Pixie's architecture gives you a secure, cost-effective and scalable way to access unlimited data, deploy [AI/machine learning] models at source and set up streaming telemetry pipelines."

Zain AsgarZain Asgar

Pixie is standardizing its observability offerings with the CNCF OpenTelemetry project standards, Asgar said. The new partnership with AWS is an expansion of a recent collaboration between AWS and New Relic on the AWS Distro for OpenTelemetry.

"Might this signal a change in the AWS approach to open source?" asked Eric Newcomer, CTO of WSO2, which offers a variety of open source software for API management and other tasks. "They have been criticized in the past for not giving back to the community. I will be very interested to see whether this means they will be more actively contributing to CNCF projects."

AWS' move is pragmatic, however. "Pure-play observability vendors such as New Relic, along with Pixie added through acquisition, are in high demand among cloud partners like AWS clamoring for technology to fill out their operational lifecycle management requirements," said Charlotte Dunlap, an analyst at GlobalData in Santa Cruz, Calif.

Next Steps

An introduction to eBPF and where it shines

Startup's eBPF APM tools turn up heat on Datadog

Dig Deeper on Software development lifecycle

Cloud Computing
App Architecture