Multi-tenancy Solutions for Kubernetes Series: Evaluation Criteria
Overview
Overview
Kubernetes' official website has a good explaination about the motivation, use cases, considerations, and the potential patterns of implementing the multi-tenancy in Kubernetes.
Running Kubernetes is costly in terms of the required resources and the overhead of managing the clusters. And Kubernetes does not support multi-tenancy natively even it has some building blocks to support isolation for different teams and users.
A real multi-tenancy solution is still desirable.
Multi-tenancy in this context is about how to share a physical Kubernetes cluster to different tenants, with some sort of isolation and control. Some may even look at a broader landscape to build multi-tenancy solution across multiple clusters.
There are many innovative solutions / open-source projects in the vibrant Kubernetes community.
In this blog series, I'm going to discuss the potential evaluation criteria and dive into two very interesting projects, among others, to walk through the evaluation process:
This blog series includes 4 parts:
- The evaluation criteria (this blog)
- The evaluation process for vcluster (will add the link once it's ready)
- The evaluation process for Capsule (will add the link once it's ready)
- The comparison and conclusion (will add the link once it's ready)
Evaluation Criteria
In general, there are two major sets of evaluating criteria:
- Control Plane Isolation
- Data Plane Isolation
Even the potential topics under these two major sets of evaluating criteria would not be exhaustive, but let's have a try and focus on some key areas. I may add more evaluation criteria along the days, if that makes sense.
I'm going to evaluate the criteria in a multi-node k3s cluster, running on Docker (powered by Rancher Desktop), on my MacBook Pro.
Control Plane Isolation
| Topic | Why it's important? |
|---|---|
| Authentication | Each tenant may want to reuse their existing user management & authentication mechanism so it may become the starting point of integration with Kubernetes cluster(s) |
| Authorization with Role-Based Access Control (RBAC) | Each tenant should have their needs of RBAC so that they can assign different teams for performing different tasks |
| Resource Quotas | Maintaining proper limitations/boundaries, like resource quotas, is very key in a multi-tenant env for necessary control and fairness |
| Cluster-wide resources (e.g. CRD, ClusterRole/ClusterRoleBinding) | Cluster-wide resources like CRDs are very common in Operator patterns for highly specialized software components and it may become a huge management challenge in a multi-tenant env |
Data Plane Isolation
| Topic | Why it's important? |
|---|---|
| Container Runtime | Containers share the host kernel, which can lead to vulnerabilities and potential container breakouts or remote code execution. To mitigate these risks, each tenant may have their own preferences to run containers on different "Runtime Class", especially when there are quite VM-like options like Kata Containers. |
| Storage Isolation | Storage can be a must-have feature when running stateful workloads in Kubernetes. Having tenant-managed StorageClass can offer the necessary power for tenants to work diversified workloads. Meanwhile, there is always a need to restrict the usage of some volumes like hostPath to avoid the abuse of node-local storage. |
| Network Isolation | Network isolation is typically achieved through NetworkPolicy, which restricts network flow between pods and reduces unexpected network access. Advanced network isolation can be achieved through service mesh implementations like Istio or Linkerd. |
Getting Ready
Install k3d CLI, and then use it to provision and simulate a multi-node Kubernetes Cluster.
1wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
2k3d cluster create mycluster --agents 3
In just 1-2 minutes, depending on your download speed, you should be able to get a cluster like this, which has 1 Master Node and 3 Worker Nodes:
1$ kubectl get nodes
2NAME STATUS ROLES AGE VERSION
3k3d-mycluster-server-0 Ready control-plane,master 17s v1.27.4+k3s1
4k3d-mycluster-agent-0 Ready <none> 13s v1.27.4+k3s1
5k3d-mycluster-agent-1 Ready <none> 14s v1.27.4+k3s1
6k3d-mycluster-agent-2 Ready <none> 12s v1.27.4+k3s1
Okay, now we're ready to evaluate our solutions.
I'll follow up accordingly and evaluate vcluster and Capsule, one by one.
So stay tuned!