A smarter way to use imagePullSecrets in Kubernetes Cluster using ServiceAccounts

Kubernetes is everywhere now a days, so does the container images and fetching the images from a private registry is a norm because of N number of reasons including security, that being the topmost.

Recap

Just to give you a heads up before we dive further, let's assume I fire kubectl create deployment nginx --image nginx what do you think happens?

1. Using kubeconfig, kubectl first authenticates with the API server.


2. It then constructs a deployment object using the values which we provided like deployment name as nginx, image to be used as nginx and some default values and then sends a request to the API server.


3. API server then validates the request and the object definition.


4. This is further stored in etcd datastore.


5. Then the controller manager specifically deployment controller comes into the picture which creates a replica set.


6. Then the replica set creates a pod (this is where our nginx image will be used).


7. Then the schedular assigns the pod to a node, where it will run based on multiple conditions. If interested please check out my previous article on what all things happens in the background: Resource management in Kubernetes
   


8. Once the node is identified, the kubelet running on the node is responsible for the management of the pod.


9. Once the pod is scheduled on a specific node, the kubelet looks for the image to be used and if it is already present.

if that is present locally, it will use it.

If not, it pulls the image from the registry.

If the registry is public, there are no issues, it just pulls the image, runs the pods, and we are done.

When imagePullSecrets are used?

• Assume the registry is private, this is where imagePullSecrets come into play.


• Kubernetes will need the secrets to authenticate the registry, so that it can pull the image.

How it works?

• Firstly, we need to create a secret which contains the credentials and some details of our docker registry like server-name, username, password, email, etc.


• It can be created both imperative as well as declarative way.
  

$ k get secrets  -o yaml dockersec

apiVersion: v1

data:

  .dockerconfigjson: ENCODED JSON OBJECT

kind: Secret

metadata:

  name: dockersec

  namespace: default

type: kubernetes.io/dockerconfigjson 

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD

• 
  Quick note: The secret must be of a type kubernetes.io/dockerconfigjson or docker-registry based on the way you choose to create the secret.


• Once the secret is created, it must be attached to a pod so that it can use it.


• Just add below content in your pod/deployment/statefulset yaml files, the name of the secret for me is dockersec but you can change based on your requirements.
  

spec:

  imagePullSecrets:

    - name: dockersec

• Before pulling the image, the kubelet checks if the image is from a private registry. If so, it looks for imagePullSecrets in the spec.


• Without proper imagePullSecrets, the image pull will fail with ImagePullBackOff.

Problem

• Now assume you have hundreds of YAML files and hundreds of Kubernetes Object where we have to use this imagePullSecrets.


• Just for the sake of doing once, you thought to do it manually but just imagine there was a change in the name of the Kubernetes secret.


• Maybe because the organization now decides to streamline the naming convention of all the Kubernetes objects based on environments like prod/preprod/staging/etc.
  
  
  
  
  • Or your org is now moving to a different registry or a different provider.
  
  
  • Or the application is getting re-factored based on different namespaces or functions.
  
  
  • There can be N number of reasons and doing all these changes again is a tricky and tiresome task.
  
  
  
  
  
  

One way to look at it is, use of helm charts which can help us to manage all the Kubernetes objects and this kind of a change can be pretty straight forward provided the helm charts are structured correctly and using values file we can just update the name of the secret and then can upgrade the chart, and we are done.

But if in case we are not using and package manager like helm and we are just working with vanilla YAML files then for that as well there is a neat way to do it.

The NEAT way!

We can attach the imagePullSecrets with the serviceaccounts and where-ever the serviceaccount is used, it will automatically populate the imagePullSecrets in the respective object like Pods.

• Below is the current status of my Kubernetes objects, no secrets, a default serviceaccount and no imagePullSecrets updated in the pod.
  

$ k get secrets

No resources found in default namespace.

$ k get sa

NAME      SECRETS   AGE

default   0         33m

$ k get pods

NAME                     READY   STATUS             RESTARTS   AGE

nginx-65fd88fc88-8xjc9   0/1     ImagePullBackOff   0          32m

• Create the docker-registry secret
  

$ k create secret docker-registry dockersec --docker-server=DOCKER_REGISTRY --docker-username=USERNAME --docker-password=PASSWORD

secret/dockersec created

$ k get secret

NAME        TYPE                             DATA   AGE

dockersec   kubernetes.io/dockerconfigjson   1      3s

• Patch the default serviceaccount to have the imagePullSecrets. It can be any serviceaccount based on your requirements.
  

$ kubectl patch serviceaccount default -p '{"imagePullSecrets": [{"name": "dockersec"}]}'

serviceaccount/default patched

$ k get sa

NAME      SECRETS   AGE

default   0         35m

• Or you can manually edit the serviceaccount as well and add below content.
  

$ k get sa default -o yaml

apiVersion: v1

imagePullSecrets:   # ADD

- name: dockersec   # ADD

kind: ServiceAccount

metadata:

  name: default

  namespace: default

• To see this in action now, delete the pod which is in failed state.
  

$ k delete pod nginx-65fd88fc88-8xjc9 --force

• Check the events of the pod, now it was successfully able to pull the image from the private registry.
  

Events:

  Type     Reason     Age                From               Message

  ----     ------     ----               ----               -------

  Normal   Scheduled  65s                default-scheduler  Successfully assigned default/nginx-65fd88fc88-w8mph to minikube

  Normal   Pulling    64s                kubelet            Pulling image "sunnybhambhani/test:v1"

  Normal   Pulled     37s                kubelet            Successfully pulled image "sunnybhambhani/test:v1" in 30.288s (30.288s including waiting). Image size: 567185619 bytes.

  Normal   Created    21s (x3 over 36s)  kubelet            Created container: test

  Normal   Started    21s (x3 over 36s)  kubelet            Started container test

• Interesting bit is imagePullSecrets are automatically added to the object.
  

$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i serviceaccountname

serviceAccountName: default

$ k get pods nginx-65fd88fc88-w8mph -o yaml | grep -i imagepullse -A 1

  imagePullSecrets:

  - name: dockersec

Bonus Tip

• In case the pod is not in running status, it can be due to multiple reasons.


• Just describe the pod to see what is happening in the background.


• kubectl describe pod POD_NAME | awk '/^Events:/ {found=1; next} found', it looks for the line starting with Events, once found prints the remaining lines.
  

$ kubectl describe pod nginx-65fd88fc88-8xjc9 | awk '/^Events:/ {found=1; next} found'

  Type     Reason     Age                     From               Message

  ----     ------     ----                    ----               -------

  Normal   Scheduled  7m21s                   default-scheduler  Successfully assigned default/nginx-65fd88fc88-8xjc9 to minikube

  Normal   Pulling    4m17s (x5 over 7m20s)   kubelet            Pulling image "REGISTRY/test:v1"

  Warning  Failed     4m14s (x5 over 7m17s)   kubelet            Failed to pull image "REGISTRY/test:v1": Error response from daemon: pull access denied for REGISTRY/test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

  Warning  Failed     4m14s (x5 over 7m17s)   kubelet            Error: ErrImagePull

  Warning  Failed     2m41s (x19 over 7m16s)  kubelet            Error: ImagePullBackOff

  Normal   BackOff    2m20s (x21 over 7m16s)  kubelet            Back-off pulling image "REGISTRY/test:v1"

• Here the logs clearly state, requested access to the resource is denied and in-turn the status of the pod is ImagePullBackOff.
  

$ k get pods

NAME                     READY   STATUS             RESTARTS   AGE

nginx-65fd88fc88-8xjc9   0/1     ImagePullBackOff   0          76s

References:

• Google.


• Kubernetes docs (https://kubernetes.io/docs/home/).


• kubectl explain.


• Linux man pages.

PS: This works with any flavor of Kubernetes: minikube, Elastic Kubernetes Service(EKS), Azure Kubernetes Service(AKS), Google Kubernetes Engine(GKE), etc.

I hope this will be helpful, feel free to add your thoughts and experiences, Happy Learning! 😄