How To Use Kubernetes Taints and Tolerations?

Kubernetes is an open-source Container Management tool that automates container deployment, container scaling, descaling, and container load balancing (also called a container orchestration tool). 

Suppose there is  a scenario, you have a single cluster where you run all of your production, staging, and testing workloads. Our Productions nodes are of higher capacity, staging nodes do not have that much capacity as production nodes and testing nodes are of lower capacity for obvious reasons. 

Let’s take another example, you want some node with GPU and you don’t want to run any other workload except that requires GPU like ML models,Because you don’t want to waste resources and money.

How can you make sure that the production pods run only on production nodes, staging pods on staging nodes, and so on?

Initially, Kubernetes supports node selectors and node affinity to achieve that and later it extends this functionality by introducing Taints and Tolerations. These are used for restricting which pod can use which node. Node selector and Node affinity are a topic for another day, we can learn about them later.

Taints allow rejecting pods from being scheduled on certain nodes by only adding taints to the node without having to modify existing pods. A pod can only be scheduled to a node if it tolerates that node’s taint. In other words, taints and tolerations are used to repel pods from certain nodes. Taints have a key, value, and an effect, and are represented as <key>=<value>:<effect>. 

Let’s learn about taint effects, there are three possible taint effects:

• NoSchedule – This effect prevents pods from being scheduled on a certain node if those pod(s) can not tolerate that node’s taint.
• PreferNoSchedule – This effect tells the scheduler to try to avoid scheduling pod(s) on the node, but can schedule it to the node if it can’t schedule those pod(s) to some other nodes.
• NoExecute – If you add a NoExecute taint to a node, pods that are already running on that node and don’t tolerate this taint will be evicted immediately from that node.

To follow this tutorial you need a multi-node cluster, it can be a kind cluster, kubeadm, or any managed Kubernetes cluster like EKS, GKE, or AKS hosted by cloud service providers like AWS, GCP, or Azure. For the sake of simplicity, I’m using a kind cluster with 3 nodes.

Setup multi-node kind cluster

1. Create a multi-node-cluster.yaml manifest file

# three nodes (two workers) cluster config
apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
  - role: worker
  - role: worker

$ kind create cluster --config=multi-nodes-cluster.yaml

2. Set k as an alias to kubectl so that we’ve to type less (Optional)

$ alias k="kubectl"

3. Check details of our cluster

$ k get no

(k is an alias to kubectl and no is shorthand for nodes)

As we can see above our cluster is ready

Taints and Tolerations

Let’s dig deep into our control-plane details,

$ k describe no kind-control-plane | grep -i taints

As we can see that the control-plane node has a single taint with the effect of NoSchedule. This taint prevents pods from being scheduled on this node unless those pod(s) can tolerate this taint.

Now let’s check taint on other worker nodes,

$ k describe no kind-worker | grep -i taints

The worker node has no taint which means any pod can schedule on it.

Adding custom taints to a node

Let’s consider our first worker node as the production node so that only production deployment pods can schedule it. To do so we need to add a taint with key node-type, value production, and with NoSchedule effect.

$ k taint no kind-worker node-type=production:NoSchedule

check the node’s taint

$ k describe no kind-worker | grep -i taints

As we can see above our first node has been tainted with the NoSchedule effect which means only pods that can tolerate this taint can schedule on it.

Let’s deploy a regular deployment with 5 replicas and see what will happen

$ k create deploy webserver --image=nginx:1.23-alpine --replicas=5

let’s check where the pods get scheduled

$ k get po -o wide

We can see all the pods are scheduled on the worker2 node because of taint on the production node. Now let’s see how we can deploy production pods on production nodes.

Adding Tolerations to Pods

To deploy production pods to production nodes they need to tolerate the production nodes’ taint. Let’s create a production deployment webserver-prod.yaml manifest file.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webserver-prod
  labels:
    env: prod
spec:
  replicas: 5
  selector:
    matchLabels:
      env: prod
  template:
    metadata:
      labels:
        env: prod
    spec:
      containers:
       - name: nginx
         image: nginx:1.23-alpine
         imagePullPolicy: IfNotPresent
      tolerations:
       - key: "node-type"
         value: "production"
         operator: "Equal"
         effect: "NoSchedule"

We add the toleration so that these production pods are scheduled on the production nodes. Let’s create this workload,

$ k create -f webserver-prod.yaml

let’s again check where the pods get scheduled

$ k get po -o wide | grep webserver-prod

As we can see that these production pods get scheduled to regular nodes as well. To make sure production pods are only scheduled to production nodes we’ve to taint the regular nodes to reject the production pods by adding key node-type, value non-production, and with NoSchedule effect.

$ k taint no kind-worker2 node-type=non-production:NoSchedule

again deploy our production deployment, before deploying Delete previous deployments

$ k delete deploy --all

$ k create -f webserver-prod.yaml

check where the pods get scheduled

$ k get po -o wide

this time all of our production pods are scheduled only on production workloads that are on a first worker node.

Remove Taints from Nodes

To remove the taint from a node we just need to append – (dash) with the effect and the taint will be removed from that node, but removing the taint from a node will not affect any running workloads it will affect only new workloads after that.

$ k taint no kind-worker node-type=production:NoSchedule-

again check the taint on the node.

$ k describe no kind-worker | grep -i taints

As we can see above the taint is removed from the node.