The pros and cons of hosting AI workloads on Kubernetes

Benefits of Kubernetes for AI

The popularity of using Kubernetes for AI workloads didn't come out of nowhere. Kubernetes offers several benefits for uses such as AI model training and inference.

Efficient resource allocation

One of the key goals of Kubernetes is to allocate CPU, memory and networking resources efficiently. It does this by dynamically deploying workloads across server clusters based on each workload's resource requirements and the available servers with sufficient resources at a given time. The aim is to avoid scenarios where some servers sit underutilized while others are maxed out on resource consumption due to inefficient workload placement.

Efficient use of resources is valuable in any context. It’s especially important for AI, given the heavy resource consumption of AI workloads.

Extensive ability to scale

Along similar lines, Kubernetes makes it easy to scale infrastructure up and down quickly -- another important benefit for AI workloads where resource requirements might change significantly over time.

For instance, while training an AI model, an AI/ML engineering team might require access to a vast amount of compute resources and data storage. During this period, it could scale up its Kubernetes cluster by adding more servers. Once training is complete, the cluster can scale back down to avoid wasting money on unnecessary resources.

It’s possible to scale infrastructure using other types of hosting platforms as well. But with Kubernetes, scaling is simple because it's a matter of adding or removing servers from a cluster.

Direct GPU access

With help from extensions like the Nvidia GPU Operator (a Kubernetes add-on that manages GPU resources within a cluster), it's relatively simple to allocate GPU access to Kubernetes-based workloads. This is not just true of workloads that run on bare-metal servers within a Kubernetes cluster, but also those hosted using VMs.

For many AI workloads, GPU access is critical because the vast parallel computing power of GPUs is essential for model training and inference at scale.

Consistency between dev/test and production environments

Organizations can use Kubernetes both to develop and test workloads, as well as to run them in production. Doing it all on the same platform offers environment parity, which is consistency across dev/test and production environments. By extension, it reduces the risk that configuration differences between environments could lead to bugs, security risks or performance issues that engineers don't catch until software is in production and available to end users.

Here again, environment parity is advantageous for most types of workloads, but it's particularly valuable when deploying highly complex AI models whose behavior can be drastically altered by configuration changes like modifications to model weights.

As an open source platform, Kubernetes poses less of a lock-in risk than most proprietary platforms.

Reduced vendor lock-in risks

As an open source platform, Kubernetes poses less of a lock-in risk than most proprietary platforms.

It's important to note that many Kubernetes distributions exist, and some include proprietary tooling or add-ons that can complicate migration between Kubernetes environments. Because of this, it would be wrong to assume that Kubernetes is totally free of vendor lock-in challenges. However, it's still more flexible than hosting AI workloads using something like a proprietary cloud service.

This flexibility is especially advantageous for businesses in the early stages of AI adoption. For example, these users can modify their AI technology or switch to different models while keeping the same underlying hosting infrastructure in place.

Drawbacks of Kubernetes for AI

Hosting AI on Kubernetes can also pose challenges. Key roadblocks include greater complexity, high overhead, add-on requirements and limited resource optimization.

Greater complexity

Kubernetes adds another layer to hosting stacks that wouldn't exist if workloads ran directly on servers, with no orchestrator in the mix. In this sense, it increases the technical complexity that engineers must contend with.

Resource overhead

Kubernetes also increases the amount of resources required to host a workload. The orchestration software consumes CPU, memory and storage, leaving fewer resources available for actual AI workloads to use.

GPU allocation requires add-ons

In most cases, Kubernetes needs add-ons to allocate GPUs, adding even more complexity for hosting AI workloads. However, organizations that run AI directly on GPU-equipped bare-metal servers would avoid this challenge.

Limited ability to optimize resource efficiency

As noted, Kubernetes can dynamically schedule workloads in a way that helps to use resources efficiently. But there are limits on how well it can do this, mainly because Kubernetes has no inherent way of knowing exactly how much CPU, memory or GPU a given workload will require.

It relies on admins to provide guidance by configuring resource allocation ranges. If the ranges are wrong (or not configured at all), AI workloads may end up underperforming due to lack of resources, or wasting money because they are over-provisioned.

Alternatives to Kubernetes for AI workload hosting

If Kubernetes isn't the best fit for an organization's AI needs, alternative infrastructure strategies include self-managed servers, serverless functions and fully managed AI services. CIOs should review all options to find the strategy that best fits their business.

Self-managed servers

AI can run directly on servers, without orchestration capabilities in the mix. This requires engineers to deploy and manage workloads manually, an approach that could work well for businesses that only have a handful of AI workloads to operate, and that don't need to scale their host infrastructure up or down frequently.

Serverless functions

AI workloads that run periodically, such as an AI agent that is only active a few times a day, are good candidates for hosting through a serverless function service. These services simplify setup and management complexity for businesses, while also enabling highly efficient resource usage. This is because the organization only pays infrastructure costs when the workload is active.

Cloud-based AI hosting services

For AI workloads that must be active continuously or near-continuously, a simple hosting option is cloud services designed specifically for operating AI models and/or agents. Examples include Amazon Bedrock and Google's Gemini Enterprise Agent Platform. These services often provide integrated AI development tooling, making them essentially platform-as-a-service offerings for AI.

Fully managed AI services

Businesses seeking to avoid the complexity of managing AI workloads entirely can opt for fully managed third-party services. With this strategy, an external vendor hosts models, agents or other workload types and makes them available to customers through web interfaces or APIs.

Chris Tozzi is a freelance writer, research adviser, and professor of IT and society who has previously worked as a journalist and Linux systems administrator.