Add new Node to k8s cluster with Bootstrap token

Use this technique to add new node to the cluster without providing any certificates and without having to restart the kube-apiserver

Suraj Deshmukh

4 minute read

Few days back I wrote a blog about adding new node to the cluster using the static token file. The problem with that approach is that you need to restart kube-apiserver providing it the path to the token file. Here we will see how to use the bootstrap token, which is very dynamic in nature and can be controlled by using Kubernetes resources like secrets.

So if you are following Kubernetes the Hard Way to set up the cluster here are the changes you should do to adapt it to run with bootstrap token.

Master Node changes


Add this flag --enable-bootstrap-token-auth=true to your kube-apiserver service file. In the end your service file should look like following:

cat <<EOF | sudo tee /etc/systemd/system/kube-apiserver.service
Description=Kubernetes API Server

ExecStart=/usr/local/bin/kube-apiserver \\
  --advertise-address=${INTERNAL_IP} \\
  --allow-privileged=true \\
  --apiserver-count=3 \\
  --audit-log-maxage=30 \\
  --audit-log-maxbackup=3 \\
  --audit-log-maxsize=100 \\
  --audit-log-path=/var/log/audit.log \\
  --authorization-mode=Node,RBAC \\
  --bind-address= \\
  --client-ca-file=/var/lib/kubernetes/ca.pem \\
  --enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\
  --enable-bootstrap-token-auth=true \\
  --enable-swagger-ui=true \\
  --etcd-cafile=/var/lib/kubernetes/ca.pem \\
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \\
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \\
  --etcd-servers= \\
  --event-ttl=1h \\
  --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \\
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \\
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \\
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \\
  --kubelet-https=true \\
  --runtime-config=api/all \\
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \\
  --service-cluster-ip-range= \\
  --service-node-port-range=30000-32767 \\
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \\
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \\



Add following flag --controllers=*,bootstrapsigner,tokencleaner to the controller manager service file. So service file should look like following:

cat <<EOF | sudo tee /etc/systemd/system/kube-controller-manager.service
Description=Kubernetes Controller Manager

ExecStart=/usr/local/bin/kube-controller-manager \\
  --address= \\
  --cluster-cidr= \\
  --cluster-name=kubernetes \\
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \\
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \\
  --controllers=*,bootstrapsigner,tokencleaner \\
  --kubeconfig=/var/lib/kubernetes/kube-controller-manager.kubeconfig \\
  --leader-elect=true \\
  --root-ca-file=/var/lib/kubernetes/ca.pem \\
  --service-account-private-key-file=/var/lib/kubernetes/service-account-key.pem \\
  --service-cluster-ip-range= \\
  --use-service-account-credentials=true \\


Bootstrap secret

The bootstrap token that we are using to setup is 07401b.f395accd246ae52d(You can generate one of yourself). This token has two parts 07401b which is public id and private part f395accd246ae52d a secret. Read more about the token, what is allowed and what it should look like here. And about the bootstrap token secret format here.

Create following secret which has certain requirements. The name of the secret should of the format bootstrap-token-<token public id> and should be available in kube-system namespace.

cat <<EOF | kubectl apply --kubeconfig admin.kubeconfig -f -
apiVersion: v1
kind: Secret
  # Name MUST be of form "bootstrap-token-<token id>"
  name: bootstrap-token-07401b
  namespace: kube-system

# Type MUST be ''
  # Human readable description. Optional.
  description: "Created for Kubernetes the Hard Way"

  # Token ID and secret. Required.
  token-id: 07401b
  token-secret: f395accd246ae52d

  # Allowed usages.
  usage-bootstrap-authentication: "true"
  usage-bootstrap-signing: "true"

RBAC policies to enable bootstrapping

The user authenticated by that token belongs to the group system:bootstrappers, that is why following permissions are given to that group.

kubectl create clusterrolebinding kubelet-bootstrap \
  --clusterrole=system:node-bootstrapper \

kubectl create clusterrolebinding node-autoapprove-bootstrap \ \

kubectl create clusterrolebinding node-autoapprove-certificate-rotation \ \

Worker node changes


Add this flag --bootstrap-kubeconfig, it is a path to kubeconfig which we will generate shortly, it contains bootstrap token and information to talk to the kube-apiserver. And also you don’t need to provide --kubeconfig but provide a path to it, this will be auto-generated and saved at that path.

Your kubelet service file should look like following:

cat <<EOF | sudo tee /etc/systemd/system/kubelet.service
Description=Kubernetes Kubelet

ExecStart=/usr/local/bin/kubelet \\
  --bootstrap-kubeconfig=/home/vagrant/bootstrap.kubeconfig \\
  --config=/var/lib/kubelet/kubelet-config.yaml \\
  --container-runtime=remote \\
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \\
  --image-pull-progress-deadline=2m \\
  --kubeconfig=/home/vagrant/kubeconfig \\
  --network-plugin=cni \\
  --register-node=true \\


Now changes in kubelet configuration file

cat <<EOF | sudo tee /var/lib/kubelet/kubelet-config.yaml
kind: KubeletConfiguration
    enabled: false
    enabled: true
    clientCAFile: "/home/vagrant/ca.pem"
  mode: Webhook
clusterDomain: "cluster.local"
  - ""
podCIDR: "${POD_CIDR}"
rotateCertificates: true
runtimeRequestTimeout: "15m"
serverTLSBootstrap: true

Provide appropriate path to clientCAFile. And add two more fields rotateCertificates: true & serverTLSBootstrap: true, this will enable cert rotation.

Create bootstrap kubeconfig

# I have used the ip address of my kube kube-apiserver use yours
kubectl config set-cluster kthwkinvolk \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server= \

# this token is above generated
kubectl config set-credentials kubelet-bootstrap \
  --token=07401b.f395accd246ae52d \

kubectl config set-context default \
  --cluster=kthwkinvolk \
  --user=kubelet-bootstrap \

kubectl config use-context default \

This is the bootstrap config file which we referred earlier in kubelet service file.

Now once you start kubelet service it will use the bootstrap token in the initial request and fetch the certificates.

Trying it all in one go

Here is the link to the gist where you can start a master and a node using Vagrant and create cluster using above setup.

Hope that helps. Happy Hacking.

comments powered by Disqus