Goal
[!NOTE]
In hurry? Jump to the result!
The goal of this document is to generate auto signed certificate for any pod with the following projected volumes:
volumes:
- name: creds
projected:
sources:
- podCertificate:
signerName: row-major.net/spiffe
keyType: ED25519
credentialBundlePath: service.crt
keyPath: service.key
- clusterTrustBundle:
name: row-major.net:spiffe:primary-bundle
path: ca.crt
Table of Contents
- cover_image: ./thumbnail.png
- Goal
- Table of Contents
Walkthrough
- Setup: Working Directory
- Setup: Kind Cluster with Cert Provisioning Enabled
- Setup: Mash Controller Deployed
- Verify: Auto Distributed Certificate Feature on Sample Deployment
- What's next?
- Closing
Walkthrough
Here is the step-by-step record of how I achieved the goal.
Setup: Working Directory
Let's quickly create a test directory build:
test_name=pod_certificate_request
tmp_dir=$(date +%y%m%d_%H%M%S_$test_name)
mkdir -p ~/test_dive/$tmp_dir
cd ~/test_dive/$tmp_dir
Setup: Kind Cluster with Cert Provisioning Enabled
[!NOTE]
Please note that the point of this blog's release already contains v1.35+ that includes the feature, so we do not need to set specific version.
Create a kind cluster locally with the following command:
_cluster_name="cert-provisioning"
_k8s_version="v1.35.0"
cat <<EOF | kind create cluster --name "$_cluster_name" --image kindest/node:$_k8s_version --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
PodCertificateRequest: true
ClusterTrustBundle: true
ClusterTrustBundleProjection: true
runtimeConfig:
"certificates.k8s.io/v1beta1/podcertificaterequests": "true"
"certificates.k8s.io/v1beta1/clustertrustbundles": "true"
nodes:
- role: control-plane
- role: worker
EOF
Check new cluster created with version v1.35.0:
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# cert-provisioning-control-plane Ready control-plane 29s v1.35.0
# cert-provisioning-worker Ready <none> 19s v1.35.0
Setup: Mash Controller Deployed
Let's deploy the mesh-controller developed by @ahmedtd as a sample. First, clone the helper project:
git clone https://github.com/ahmedtd/mesh-example.git mesh_example
Before we deploy the mesh-controller, we need to create a CA pool secret that the controller can use to sign the certificate requests. The mesh-controller expects two different CA pool secrets:
service-dns-ca-pool: for service DNS certificates between kubeapi server and mesh controller
spiffe-ca-pool: for ca certificate to be used to sign the auto distributed certificates to pods
The helper project provides the following command to create the CA pool & save as k8 secrets:
(cd mesh_example && go run ./cmd/meshtool make-ca-pool-secret --namespace mesh-controller --name service-dns-ca-pool --ca-id 1)
(cd mesh_example && go run ./cmd/meshtool make-ca-pool-secret --namespace mesh-controller --name spiffe-ca-pool --ca-id 1)
Check secrets created:
kubectl get secrets -n mesh-controller
# NAME TYPE DATA AGE
# service-dns-ca-pool Opaque 1 9s
# spiffe-ca-pool Opaque 1 9s
With secrets available as prerequisite, we can deploy the mesh controller:
kubectl apply -f ./mesh_example/controller-manifests
# namespace/mesh-controller configured
# clusterrole.rbac.authorization.k8s.io/meshtool-signer created
# clusterrolebinding.rbac.authorization.k8s.io/meshtool-is-a-meshtool-signer created
# role.rbac.authorization.k8s.io/meshtool-controller created
# rolebinding.rbac.authorization.k8s.io/meshtool-is-a-meshtool-controller created
# deployment.apps/mesh-controller created
The command above will create a namespace mesh-controller and deploy the mesh-controller in it. mesh-controller is a custom controller and its current job is to:
- Watch
PodCertificateRequestresources, created by Kubernetes's kubelet with user's deployment's request - Sign the certificate request with desiginated CA pool injected as a secret
- Return the signed certificate and key back to the kubelet
Check if the mesh controller is running:
kubectl get pods -n mesh-controller
# NAME READY STATUS RESTARTS AGE
# mesh-controller-547cd7996b-blqsf 1/1 Running 0 20s
Verify: Auto Distributed Certificate Feature on Sample Deployment
With all the prerequisites and the mesh controller in place, it's time to put it to the test. Let's deploy a sample application and verify if the certificates are automatically distributed and correctly mounted inside the pod.
First of all, let's create a namespace for our sample application:
kubectl create namespace podcert-demo
# namespace/podcert-demo created
Next, deploy a k8s deployment:
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: spiffe-demo-deploy
namespace: podcert-demo
spec:
replicas: 2
selector:
matchLabels:
app: spiffe-demo
template:
metadata:
labels:
app: spiffe-demo
spec:
serviceAccountName: default
automountServiceAccountToken: false
containers:
- name: app
image: alpine/openssl
command: ["sh", "-lc", "sleep infinity"]
volumeMounts:
- name: creds
mountPath: /var/run/tls
readOnly: true
volumes:
- name: creds
projected:
sources:
- podCertificate:
signerName: row-major.net/spiffe
keyType: ED25519
credentialBundlePath: service.crt
keyPath: service.key
- clusterTrustBundle:
name: row-major.net:spiffe:primary-bundle
path: ca.crt
EOF
# deployment.apps/spiffe-demo-deploy created
Note that the kubelet can notice that this deployment (or its pods) require the certificate by:
- name: creds
projected:
sources:
- podCertificate:
signerName: row-major.net/spiffe
keyType: ED25519
credentialBundlePath: service.crt
keyPath: service.key
- clusterTrustBundle:
name: row-major.net:spiffe:primary-bundle
path: ca.crt
The mesh-controller by default is watching the signerName row-major.net/spiffe and row-major.net/service-dns. If we ever build our own signer, we can simply change the singerName in the deployment manifest.
Finally, let's check the certificate and key are mounted on /var/run/tls inside the pod:
kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo -- ls -al /var/run/tls
# total 4
# drwxrwxrwt 3 root root 140 Apr 6 02:46 .
# drwxr-xr-x 1 root root 4096 Apr 6 02:46 ..
# drwxr-xr-x 2 root root 100 Apr 6 02:46 ..2026_04_06_02_46_52.3629103651
# lrwxrwxrwx 1 root root 32 Apr 6 02:46 ..data -> ..2026_04_06_02_46_52.3629103651
# lrwxrwxrwx 1 root root 13 Apr 6 02:46 ca.crt -> ..data/ca.crt
# lrwxrwxrwx 1 root root 18 Apr 6 02:46 service.crt -> ..data/service.crt
# lrwxrwxrwx 1 root root 18 Apr 6 02:46 service.key -> ..data/service.key
Check the certificate auto distributed:
kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo \
-- openssl x509 -in /var/run/tls/service.crt -noout -text
# Certificate:
# Data:
# Version: 3 (0x2)
# Serial Number:
# 02:20:72:93:62:05:37:22:fe:41:4f:ad:3f:ce:c5:bd:54:31:9e:0c
# Signature Algorithm: ED25519
# Issuer:
# Validity
# Not Before: Apr 6 02:44:52 2026 GMT
# Not After : Apr 7 02:44:52 2026 GMT
# Subject:
# Subject Public Key Info:
# Public Key Algorithm: ED25519
# ED25519 Public-Key:
# pub:
# 0e:bb:f2:f7:fc:ee:5f:21:83:c3:87:da:3c:eb:79:
# dd:fe:5c:41:7e:90:cb:aa:e0:96:a8:64:e0:c5:1c:
# 11:90
# X509v3 extensions:
# X509v3 Key Usage: critical
# Digital Signature
# X509v3 Extended Key Usage:
# TLS Web Client Authentication, TLS Web Server Authentication
# X509v3 Basic Constraints: critical
# CA:FALSE
# X509v3 Subject Alternative Name: critical
# URI:spiffe://cluster.local/ns/podcert-demo/sa/default
# Signature Algorithm: ED25519
Let's see the root CA with one year validity:
kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo \
-- openssl x509 -in /var/run/tls/ca.crt -noout -text
# Certificate:
# Data:
# Version: 3 (0x2)
# Serial Number:
# 70:f5:36:4e:2b:6f:9e:fc:fb:2d:cf:5d:32:77:65:a5:fe:5e:08:64
# Signature Algorithm: ED25519
# Issuer:
# Validit
# Not Before: Apr 6 02:39:27 2026 GMT
# Not After : Apr 6 02:39:27 2027 GMT
# Subject:
# Subject Public Key Info:
# Public Key Algorithm: ED25519
# ED25519 Public-Key:
# pub:
# 87:6d:54:67:8f:0c:d7:ed:5a:e4:a5:fd:a0:fb:81:
# 7d:e2:7e:84:44:dc:5f:2f:99:12:63:b4:08:2c:b0:
# 0b:5b
# X509v3 extensions:
# X509v3 Key Usage: critical
# Digital Signature, Certificate Sign
# X509v3 Basic Constraints: critical
# CA:TRUE
# X509v3 Subject Key Identifier:
# A0:96:CE:C1:73:8E:63:5A:26:C2:0B:BE:45:46:4D:25:50:C4:7E:EB
# Signature Algorithm: ED25519
This is it! The kubelet automatically created a PodCertificateRequest and the mesh-controller signed it and returned the signed certificate and key back to the kubelet. The kubelet then mounted the certificate and key as a projected volume in the pod.
What's next?
The mesh-controller sample is a really good starting point to understand how the PodCertificateRequest works. However, if we want to generate a certificate signed by external CA, how can we do this? The next step is to create a controller that does detect the PodCertificateRequest, and instead of signing by itself with secret injected CA, it forwards the request to the external authentication/authorization service (like Athenz), and returns the signed certificate and key back to the kubelet.
Closing
If you enjoyed this deep dive, please leave a like & subscribe for more!
Also, leave comments if you have any questions or suggestions. Thank you in advance!
