How to approach Kubernetes multi-tenancy for resource isolation
Multi-tenancy in Kubernetes can partition IT resources and operations, but it can be tricky to determine just how much isolation is necessary.
It would be comforting to think of all hybrid cloud application design and cloud-native development efforts as a unifying process. Data centers and clouds are absorbed into a pool of resources, and management and orchestration are handled by a mighty single scheduler -- Kubernetes.
The problem is that IT applications have never been unified. Some applications are always more important than others, and there are reasons to focus resources on certain things at certain times. There are reasons, too, to restrict access. Even within a company, it's sometimes necessary to consider different subsidiaries, departments or applications as independent tenants. That's what makes Kubernetes multi-tenancy important.
Kubernetes supports labels, taints and tolerations as a means to recognize the need to direct resources explicitly, either to preserve expensive or scarce features on servers or to ensure that key applications get what they need, when they need it. Kubernetes multi-tenancy is a further refinement of the not-all-applications-are-equal approach. But remember, Multi-tenancy is not a substitute for network and API security.
Consider the routes that lead to multi-tenancy
Conceptually, there are two pathways toward Kubernetes multi-tenancy. The first is where a Kubernetes deployment involves tenant-specific clusters but is managed as a whole. To Kubernetes, this is federation. The other path is where clusters themselves host multiple independent tenants. Kubernetes considers this second approach multi-tenancy, but organizations should evaluate the benefits and challenges of each path.
This article is part of
What is container management and why is it important?
An enterprise might consider multi-tenancy when its IT organization supports functionally independent businesses using collective resources or plans to consolidate separate business-specific IT resources. Multi-tenancy might be beneficial to support separate subsidiaries, for example, or in a merger or acquisition between two companies with distinct IT platforms. In these cases, multi-tenancy keeps the resources and operations of separate organizations partitioned from each other.
The primary disadvantages of multi-tenancy are a loss of operations efficiency and server or cloud economies of scale. As the degree of separation among tenants increases, operations personnel and resource pools must be assigned to specific tenants. It's possible operations staff won't be able to cover all applications and resources, and the hosting efficiency of all tenants is reduced if applications can't migrate freely across all suitable hosts.
In short, multi-tenancy is a remedy that must be justified. An organization that doesn't need applications and resources partitioned from each other shouldn't even consider it. A business that does need to partition resources should carefully examine the reasons for doing so. If it's to ensure proper resource allocation, don't go beyond labels, taints and tolerations unless they've proven to be ineffective. Where that's the case, or where compliance or security requirements demand stronger partitioning of applications and resources, the next step depends on the organization.
Large enterprises, because of the vast amount of IT resources they consume and the size of their operations staff, are less likely to find these disadvantages serious enough to discourage them from separating subsidiaries or business units as independent tenants. Breaking down human and hosting resources by tenant will leave each tenant large enough to justify efficient operations and resources support. A sizable organization will therefore want to explore whether to adopt multi-tenant clusters or tenant-specific clusters under overall policy control. A mid-sized business will have to more carefully weigh the risks and benefits of cluster multi-tenancy.
Evaluate federation and clusters
Kubernetes federation allows an organization to unify multiple Kubernetes deployments and clusters. This feature is valuable for multi-cloud and hybrid cloud users that have managed Kubernetes cloud services in addition to on-premises Kubernetes. Organizations can also use this feature to create a policy umbrella over tenant-specific Kubernetes deployments. IBM, Red Hat, VMware and Google offer enhanced Kubernetes packages designed for federation.
The biggest advantage to federation-based multi-tenancy is that it offers complete isolation of the organizations and tenants, and still preserves the option to set policies for resource use and deployment of containers where needed. This is a particularly handy way to prevent semi-autonomous Kubernetes tenants from hogging shared resources or overusing cloud resources. However, each tenant must manage a Kubernetes domain to gain these benefits, which significantly increases the operations burden. It also requires that clusters be dedicated to a single tenant, which may create resource efficiency issues if some tenants are small, meaning they don't require many server resources or have highly variable server needs.
For smaller businesses -- and large businesses that can't justify federation's downsides -- cluster multi-tenancy is perhaps a better option. The key here is to correctly use the built-in cluster multi-tenancy features of Kubernetes. Assign each tenant a separate namespace, protect Kubernetes APIs and then set policies to override the normal Kubernetes rule that all components/containers/pods can communicate with each other. From there, use the Resource Quotas feature to allocate capacity by namespace; labels, taints and tolerations will also work, but they'll be harder to administer across namespaces. Finally, consider some Kubernetes add-ons or implementations to partition deployments at the operations level. This will approach federation-based multi-tenancy in terms of isolation, but with less operations overhead.
The downside of this strategy is that you get close to federation-based multi-tenancy only in terms of tenant isolation, and, in the process, create an unusual Kubernetes model in terms of communication among pods. Organizations will have to integrate any enterprise-wide applications across tenants outside Kubernetes -- a task that's expected in federation, but not within a single Kubernetes domain.
In addition, ensure operations personnel understand the differences in their multi-tenant deployment, because, when you revert to familiar practices, you'll break things -- often badly enough to compromise isolation or stability.
There's a multi-tenancy group within the Kubernetes community, with documents open only to its members. This group seems focused on hardening the current cluster-based multi-tenancy process, as a suitable replacement for the federation approach. If that's true, then multi-tenancy will likely focus on tenants that need a lot of separation, as they would if they were independent companies. Ordinary users will then need to think even harder before they attempt multi-tenancy themselves.