This blog is a recap of my old blog “Add new node to Kubernetes cluster with bootstrap token”. Like the aforementioned blog, we will look at how to enable TLS bootstrapping on an existing Kubernetes cluster at control plane level and add a new node (or modify existing ones) to the cluster using bootstrap tokens. At the end of this blog, you will learn what specific steps to take to enable TLS bootstrapping on any custom-built Kubernetes cluster.
Textual
1. Kubernetes Cluster
If you have a cluster running, you can skip this step. To demonstrate how TLS bootstrap tokens work, I have brought up a cluster using Kubernetes the Hard Way (Vagrant). This section shows how to bring up a multi-node, multi-worker Kubernetes cluster quickly.
1.1. Setup Machine
I am running this setup on a Ubuntu 20.04 machine. Install virtual-box and vagrant by executing the following commands as the root user:
apt-get update && \
apt-get -y upgrade && \
apt-get install -y make git byobu linux-generic
systemctl reboot
apt-get install -y linux-headers-`uname -r` && \
apt install -y virtualbox && \
apt-get install -y vagrant && \
dpkg-reconfigure virtualbox-dkms
1.2. Clone the Repository
mkdir -p git/kubernetes-the-hard-way-vagrant
git clone https://github.com/kinvolk/kubernetes-the-hard-way-vagrant git/kubernetes-the-hard-way-vagrant
cd git/kubernetes-the-hard-way-vagrant
1.3. Install the Cluster
./scripts/install-tools
./scripts/setup
1.4. Verify the Cluster Installation
kubectl get nodes
2. Prepare Control Plane
2.1. Setup RBAC
When a new node tries to authenticate to the control plane, the only credential used is a bootstrap token. Such a node client (kubelet) is a member of group system:bootstrappers, we need to ensure that such a client has permissions to create Certificate Signing Requests (CSRs), get them approved, renew them, etc. The following ClusterRoleBindings will ensure that these nodes join without authorisation issues. This setting needs to be done only once and does not change from node-to-node.
Run the following command to create the bindings that will allow the new nodes to connect to the cluster:
cat <<EOF | kubectl apply -f -
# Enable bootstrapping nodes to create CSR.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: create-bootstrapping-csr
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:node-bootstrapper
apiGroup: rbac.authorization.k8s.io
---
# Approve all CSRs for the group "system:bootstrappers".
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: approve-csrs
subjects:
- kind: Group
name: system:bootstrappers
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
apiGroup: rbac.authorization.k8s.io
---
# Approve renewal CSRs for the group "system:nodes".
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: approve-node-renewals
subjects:
- kind: Group
name: system:nodes
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
apiGroup: rbac.authorization.k8s.io
EOF
2.2. Modify Kubernetes Apiserver
Add the following set of flags to the kube-apiserver:
--authorization-mode=Node,RBAC
--enable-bootstrap-token-auth=true
Above setting needs to be performed on each control plane node.
2.3. Modify Kubernetes Controller Manager
Add the following flag to the kube-controller-manager:
--controllers=*,tokencleaner,bootstrapsigner
Above setting needs to be performed on each control plane node.
3. Modify / Add Workers
3.1. Create the Bootstrap Token
Please note these tokens, we will need them in two places. One, while creating a bootstrap-token embedded Kubernetes Secret object and another in the bootstrap kubeconfig for the Kubelet:
TOKEN_ID=$(openssl rand -hex 3)
TOKEN_SECRET=$(openssl rand -hex 8)
3.2. Create Bootstrap Token Secret
Kubernetes apiserver will use this secret to identify the node that is joining the cluster. Execute the following command to create the secret embedding the previously generated tokens:
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
type: bootstrap.kubernetes.io/token
metadata:
name: bootstrap-token-$TOKEN_ID
namespace: kube-system
stringData:
description: "Manually generated bootstrap token."
token-id: "$TOKEN_ID"
token-secret: "$TOKEN_SECRET"
usage-bootstrap-authentication: "true"
usage-bootstrap-signing: "true"
EOF
3.3. Create Bootstrap Kubeconfig for Kubelets
Create bootstrap Kubeconfig for Kubelet(s):
SERVER=$(kubectl config view -ojsonpath='{.clusters[0].cluster.server}')
CA_CERT=$(kubectl config view --flatten -ojsonpath='{.clusters[0].cluster.certificate-authority-data}')
cat <<EOF | tee kubeconfig
apiVersion: v1
kind: Config
clusters:
- name: local
cluster:
server: "$SERVER"
certificate-authority-data: "$CA_CERT"
users:
- name: kubelet
user:
token: "$TOKEN_ID.$TOKEN_SECRET"
contexts:
- context:
cluster: local
user: kubelet
EOF
Keep this file around it will be needed to connect worker(s) node using bootstrap token to control plane.
3.4. Modify Kubelet
Copy the bootstrap kubeconfig created in step 3.3 to the worker node you want to connect to control plane using bootstrap tokens. Save the kubeconfig on the node and provide the path to that file to flag --bootstrap-kubeconfig. The path supplied to --kubeconfig will have newly downloaded kubeconfig once the kubelet authenticates with the kube-apiserver.
Add the following set of flags to the kubelet:
--kubeconfig=/var/lib/kubelet/kubeconfig
--bootstrap-kubeconfig=/etc/kubernetes/kubeconfig
--rotate-certificates
NOTE: You can remove the file in /var/lib/kubelet/kubeconfig, especially if you modify an existing worker node, to ensure that Kubelet downloads the new kubeconfig at that location.
You can have a separate token for each worker node to reduce the attack surface. To generate distinct token repeat steps 3.1 to 3.4 for each worker node. But if you wish to use the same token for each worker node, repeat step 3.4 on each worker.
4. Verify
Ensure that the node has joined the control plane using TLS bootstrap token by executing the following command:
kubectl get nodes
Video
Watch this video to understand the same process explained in this blog. Unlike my previous blog videos, I am narrating, so you know what is going on.