In the previous blog, we discussed how any user without RBAC access to a Kubernetes secret can use a trick to access that secret. To mitigate that problem, we will use a validating admission webhook. But before looking at what sorcery this validating admission webhook server is, let us understand how Kubernetes handles the API requests.
What are admission controllers?
All requests going to the Kubernetes API server go through the following four steps:
- Authentication
- Authorisation
- Admission controllers
- Etcd Server
Image Source: Controlling Access to the Kubernetes API
Request denied at any step does not go to the next steps. Any request reaching the “Admission Controllers” has already been authenticated and authorised. There are the standard set of admission controllers shipped with the Kubernetes API server. You can enable or disable them using flags to the Kubernetes API server.
As the name suggests, the job of the admission controller is to perform certain business logic based on the request. For example, the NodeRestriction admission controller’s role is to restrict a node’s (Kubelet’s) access on top of RBAC restrictions. When Kubelet requests secrets from the Kubernetes API server, the NodeRestriction admission controller ensures that Kubelet does not request secrets for pods, not on its node. This is especially helpful if a node is compromised, then the hacker cannot request all secrets from the cluster.
What is a Validating Admission Webhooks?
For our situation, we are going to use an admission controller called ValidatingAdmissionWebhook. This allows us to dynamically load any new webserver with logic to process the requests sent by the ValidatingAdmissionWebhook admission controller. The expectation from these dynamically loaded webservers is to only respond with yes or no.
This admission controller sends a webhook to the specified endpoints when a given condition is met. We can ask things like send all the requests to a particular URL after encountering a resource called pods
of apiVersion v1
when someone is trying to CREATE
them. Now all the pod create requests will be sent to your webserver.
Image Source: A Guide to Kubernetes Admission Controllers
Kubernetes Secret Access Validator
With all that essential knowledge, we will employ it to stop an attacker from gaining access to the secret they don’t have RBAC access to. We will create a simple webhook server that will intercept all the Pod, CronJob, DaemonSet, Deployment, Job, ReplicaSet, ReplicationController and Statefulset create and update requests. On these requests, we will first figure out who (user/group) is making the request and what kind of secrets they are asking for access to?
Once we have these two pieces of information, we will do a programmatic equivalent of the following kubectl command:
kubectl --as=<API requesting user> can-i get secret <secret being accessed>
If the answer to the above is no, we will reject the request to the user.
Validating Webhook Server in Action
Install the Kubernetes Secret Access Validator by following the steps mentioned here. Now let us try to recreate the hack situation mentioned in the previous blog.
Create a test user that has access to pods, but not to secrets:
kubectl create role pod-all --verb=* --resource=pods --resource=pods/exec
kubectl create rolebinding pod-all:nastyuser --role=pod-all --user=nastyuser
kubectl create secret generic supersecret --from-literal data=supersecretvaluesinhere
alias kubectl='kubectl --as=nastyuser'
kubectl auth can-i --list
Save this pod config in a pod.yaml
, and we will try to deploy it:
apiVersion: v1
kind: Pod
metadata:
labels:
run: sleep
name: sleep
spec:
containers:
- args:
- sleep
- infinity
image: fedora
name: sleep
resources: {}
volumeMounts:
- name: hackedsecret
mountPath: /access-secret/
dnsPolicy: ClusterFirst
restartPolicy: Always
volumes:
- name: hackedsecret
secret:
secretName: supersecret
Now trying to deploy it:
$ kubectl create -f pod.yaml
Error from server: error when creating "pod.yaml": admission webhook "validating.suraj.io" denied the
request: User "nastyuser" does not have access to the secret "supersecret" in the namespace "default".
In Conclusion
This project is in a nascent stage. Right now, it only supports pod creation and update. The support for other pod creating controllers will be added soon.
I haven’t fully grasped the side effect of this webhook server running inside an operational cluster. But once I do that, I will learn the pros and cons of this approach. Ideally, this should be done at the Kubernetes API server level.
If you are aware of any other solution, please comment or let me know on Twitter.