The llm-d project, now ready for production, aims to do for distributed inference on GPUs what Kubernetes did for web apps: make the most of infrastructure resources.
By
Beth Pariseau, Lead Editor, Infrastructure News
Published: 20 Oct 2026
An open source utility that automatically distributes AI inference workloads across GPU clusters has reached production readiness this week with backing from IBM, Red Hat, Google, Nvidia and more.
These backers tout the Cloud Native Computing Foundation (CNCF) project, llm-d, as an equalizer that will allow more companies – including cloud providers beyond the big 3 hyperscalers – to host AI inference workloads by squeezing as much performance as possible from scarce, costly GPU hardware.
The cluster-level orchestrator automatically routes workloads to server endpoints to to make the most efficient use of available servers. The framework also ensures that AI workloads draw data from local caches as frequently as possible – a high cache-hit ratio – further reducing the token cost of high-scale AI inference operations.
One early adopter of the project, cloud provider DigitalOcean, uses its own forked version of llm-d to back its Managed Agents and Serverless Inference services. Without it, hosting such workloads would be prohibitively expensive, said Salman Paracha, senior vice president of AI Engineering for the company.
"We don't think we can get to distributed scale without llm-d, so it's about capability for us first, as we spread out token volume across a very large fleet of GPUs," Paracha said. "The distribution of that token workload is essential for us to achieve a higher cache-hit ratio, which results in better tokens-per-second [performance], and [ensure] a good quality of service."
The road to llm-d 1.0
The project was founded in 2025, and Red Hat already packages llm-d, along with the node-level vLLM inference engine, within its Red Hat AI product. Google also already runs those two utilities within its Google Kubernetes Engine service. While Kubernetes isn't required to run llm-d, the orchestrator integrates deeply with Kubernetes components under the hood when users choose to run them together, including the Kubernetes API Gateway Inference Extension.
This week, the open source version of the utility reached version 1.0, which means it is officially mature enough for production use. The production-ready version includes new features for multi-tenancy support that maintains quality of service among multiple AI models hosted in shared clusters, non-disruptive version upgrades and a stable, hardened API and request path. This week's release also supports optimizing AI agent workloads to save tokens by ensuring they use a local context cache rather than recomputing outputs, a feature called agent-aware serving.
Jason Andersen
Enterprises are most likely to acquire llm-d in packaged form from a cloud provider or software vendor, said Jason Andersen, an analyst at Moor Insights & Strategy. But as the current generation of GPUs ages, such a utility could also extend the longevity of existing hardware, either for enterprises that eventually embrace private AI or for a secondary market of GPU hosting providers.
"We haven't reached the point where you don't see 100% improvement in inference [performance] year over year," Andersen said. "But long-term, when that number shrinks, people might be like, 'I can run my AI workload a generation or two back and be fine with it,' which opens the door for something like llm-d."
Llm-d awaits its 'Kubernetes moment'
IBM Research used llm-d to run AI agents in support of a recent quantum computing project and also tested its cost-effectiveness. The project used llm-d to boost the performance of a cheaper open-weight model, z.AI's GLM-5.3 on its AI factory and evaluated its costs against serving a frontier model, Anthropic's Claude Opus 5, on the same system. IBM researchers wrote in a report on the project results that llm-d improved the cheaper model's performance enough to keep up with the closed frontier model.
What Kubernetes was for workload management, the llm-d layer is for models in some ways.
Ruchir Puri, Chief scientist, IBM Research
Running agents for 24 hours using Opus cost $300, while the same period on GLM-5.3 cost $26, according to Ruchir Puri, chief scientist at IBM Research, in an interview.
"What Kubernetes was for workload management, the llm-d layer is for models in some ways," Puri said. "It insulates not-so-savvy users from what models to run, how to run them, how to scale them and how to load balance, which interestingly is part of Kubernetes as well."
To be truly Kubernetes-like, llm-d would also have to offer portability among various infrastructures. Google's support for llm-d to schedule TPU workloads alongside Nvidia GPUs could signal the beginnings of such a capability, Andersen said.
"If Google can say, 'Hey, now it's easier to move off of Nvidia, which is where everybody started out, and get onto our TPUs,' there's a huge incentive for Google to make this work in the cloud," he said.
However, Paracha and Puri expressed doubts that transferring workloads between different chip architectures would be feasible at this point.
"The kernels that you have for models are very much tied into [what's] provided by, let's say, a TPU or an Nvidia machine or AMD, and then you have componentry like vLLM or SGLang, that optimizes for cache," Paracha said. "Those components are designed for high memory bandwidth scenarios, or high compute scenarios -- it's not super interchangeable today."
Another signature characteristic of Kubernetes is its ubiquity – five years after its introduction in 2014, it had become the industry standard for container management. The same can't yet be said for llm-d, though it counts Red Hat, IBM, Google Cloud, CoreWeave and Nvidia among its founding contributors and garnered public endorsements for version 1.0 this week from AMD, Cohere, JetBrains, Mistral and Roblox in addition to DigitalOcean.
"A player that you'd want to see get involved here who's pretty serious about integrating with Kubernetes is VMware with its private data center [software]," Andersen said.
Andersen also questioned whether the depth of support from vendors such as CoreWeave and Nvidia is enough to make it a full-fledged standard yet. CoreWeave offers llm-d support as an option on its cloud but hasn't made it a core part of its products, Andersen said. Nvidia has contributed aspects of its Dynamo distributed inference framework to llm-d, such as its NIXL data transfer and KV-cache management utilities, but there's significant feature overlap between the two open source orchestration tools.
Vendor support isn't necessarily the same as commitment, according to Andersen.
"They're just basically saying, 'Hey, if you deploy it on us and use it, we're cool with it,'" Andersen said. "That's all they're really saying at this point."
Beth Pariseau, senior news writer for Informa TechTarget, is an award-winning veteran of IT journalism. Have a tip? Email her or connect on LinkedIn.
Jason Andersen
