Add new Node to k8s cluster with cert rotation

Use this technique to add node to the cluster without providing any certificates

Suraj Deshmukh

3 minute read

The setup here is created by following Kubernetes the Hard Way by Kelsey Hightower. So if you are following along in this then do all the setup till the step Bootstrapping the Kubernetes Worker Nodes. In this just don’t start the kubelet, start other services like containerd and kube-proxy.

master node

Following the docs of TLS Bootstrapping, let’s first create the token authentication file. Create a file with following content:

$ cat tokenfile 

You should create the token which is as random as possible by running following command:

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

Now we need to tell the kube apiserver about this file. So add following flag to the kube-apiserver service file, with the path to above token file.


So my kube-apiserver service file looks like following:

$ cat /etc/systemd/system/kube-apiserver.service
Description=Kubernetes API Server

ExecStart=/usr/local/bin/kube-apiserver \
  --advertise-address= \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address= \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --enable-swagger-ui=true \
  --etcd-cafile=/var/lib/kubernetes/ca.pem \
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
  --etcd-servers= \
  --event-ttl=1h \
  --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
  --kubelet-https=true \
  --runtime-config=api/all \
  --service-account-key-file=/var/lib/kubernetes/service-account.pem \
  --service-cluster-ip-range= \
  --service-node-port-range=30000-32767 \
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
  --token-auth-file=/home/vagrant/tokenfile \


Once this is done, just run following commands to restart the apiserver:

sudo systemctl daemon-reload
sudo systemctl restart kube-apiserver
sudo systemctl status kube-apiserver

Now that api-server has restart we need to give permissions so that our worker node can ask for certs automatically, for that run following commands:

kubectl create clusterrolebinding kubelet-bootstrap \
    --clusterrole=system:node-bootstrapper \

kubectl create clusterrolebinding node-client-auto-approve-csr \ \

kubectl create clusterrolebinding node-client-auto-renew-crt \ \

worker node

First create a bootstrap kubeconfig file that will be used by kubelet. Run following commands to create it.

# I have used the ip address of my api-server use yours
kubectl config set-cluster kthw \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server= \

# this token is above generated
kubectl config set-credentials kubelet-bootstrap \
  --token=02b50b05283e98dd0fd71db496ef01e8 \

kubectl config set-context default \
  --cluster=kthw \
  --user kubelet-bootstrap \

kubectl config use-context default \

Now that we have this file, add following flags to the kubelet systemd service file.


Provide path to the bootstrap.kubeconfig you have generated just before. And even if you don’t have kubeconfig still provide some path where kubelet has permission to write. kubelet will create this file for you.

So my kubelet, systemd file looks like following:

$ cat /etc/systemd/system/kubelet.service
Description=Kubernetes Kubelet

ExecStart=/usr/local/bin/kubelet \
  --bootstrap-kubeconfig=/home/vagrant/bootstrap.kubeconfig \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --container-runtime=remote \
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
  --image-pull-progress-deadline=2m \
  --kubeconfig=/home/vagrant/kubeconfig \
  --network-plugin=cni \
  --register-node=true \
  --rotate-certificates=true \
  --rotate-server-certificates=true \


And KubeletConfiguration looks like this:

$ cat /var/lib/kubelet/kubelet-config.yaml
kind: KubeletConfiguration
    enabled: false
    enabled: true
    clientCAFile: "/var/lib/kubernetes/ca.pem"
  mode: Webhook
clusterDomain: "cluster.local"
  - ""
podCIDR: ""
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"

Once this is done, just run following commands to restart the kubelet:

sudo systemctl daemon-reload
sudo systemctl restart kubelet
sudo systemctl status kubelet

master node

List the request that node has made:

$ kubectl get csr
NAME                                                   AGE   REQUESTOR           CONDITION
node-csr-WXnon3AEhdxgZ1FZ2reqKRQmWS-pwP3x263YbwUAH9k   11m   kubelet-bootstrap   Pending

Approve that request:

$ kubectl certificate approve node-csr-WXnon3AEhdxgZ1FZ2reqKRQmWS-pwP3x263YbwUAH9k approved

Now you can see that the node has been created and you can list by running:

$ kubectl get nodes
node   Ready    <none>   27s   v1.12.0
comments powered by Disqus