We Can Resize Pods without Restarts! Or Can't We?

Kubernetes v1.27 released in April 2023 came with an exciting announcement - we can now resize pod CPU and memory requests and limits in-place! Without deleting the pod or even restarting the containers!

This happened more than a year ago and since then a lot of folks seem to think this feature is already publicly available or is due to become so tomorrow.

But the reality is that this was originally released as an Alpha feature and since then had no success moving to Beta due to a number of unresolved issues.

Latest status as of June 2024 is that it has been pushed back to v1.32:

Here's the link to that comment on Github.

So first of all - this isn't coming tomorrow. But we can still play with the feature and understand its advantages and shortcomings. Which is exactly what I'm planning to do in this post.

Get a Cluster with Alpha Features

k3d is irreplaceable when we want quickly and cheaply test Kubernetes Alpha features. All we need to do is to pass the correct feature gate to the correct control plane component.

Install k3d

If you still haven't done so - install k3d:

with curl and bash:

curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash

or with another method of your choice listed here

In our case the component is the API server and the feature gate is called InPlacePodVerticalScaling as can be seen here

I'm spinning up a single-node cluster with the following config:

cat <<'EOF' | k3d cluster create -c -

apiVersion: k3d.io/v1alpha3

kind: Simple

name: pod-resize

servers: 1

image: rancher/k3s:v1.30.2-k3s2

options:

  k3d:

    disableLoadbalancer: true

  k3s:

    extraArgs: # the feature gate is passed here

      - arg: --kube-apiserver-arg=feature-gates=InPlacePodVerticalScaling=true

        nodeFilters:

          - server:*

EOF

Now let's create a pod with one container defining resource requests and limits.

apiVersion: v1

kind: Pod

metadata:

  name: stress

spec:

  containers:

  - image: progrium/stress

    args: ["--cpu", "1", "--vm", "1", "--vm-bytes", "128M", "--vm-hang", "3"]

    name: stress

    resources:

      requests:

        memory: 100M

        cpu: 100m

      limits:

        memory: 100M

        cpu: 100m

You can create the pod with:

kubectl apply -f https://raw.githubusercontent.com/perfectscale-io/inplace-pod-resize/main/guaranteed.yaml

I'm using progrium/stress and setting it up for failure by intentionally requesting more memory than I allocate.

stress --vm 1 --vm-bytes 128M --vm-hang 3 - this tells stress to spawn one worker that allocates 128 Mb of memory and then releases them every 3 seconds.

My pod is only currently allowed to have 100M of memory, so I expect it to get killed.

While this 'stress --cpu 1 tells the container to use one whole CPU. While it's actually allowed to only use 0.1 CPU.

Quite expectedly the container gets killed the moment it starts allocating memory:

kubectl get pod

NAME     READY   STATUS      RESTARTS   AGE

stress   0/1     OOMKilled   0          7s

And it will continue restarting and getting OOMkilled until we update its memory limits. So let's save it from this misery by giving it the memory it needs:

kubectl patch pod stress -p '{"spec" : { "containers" : [{"name" : "stress", "resources": { "limits": {"cpu":"100m","memory":"150M"}}}]}}'

Oops! That didn't work!

We're getting:

The Pod "stress" is invalid: metadata: Invalid value: "Guaranteed": Pod QoS is immutable

So what we now know is that while we can change the values of limits and requests - we can't change the pod QoS class. I.e the relationship between the requests and the limits has to stay the same.

Let's try to update both the request and the limit:

kubectl patch pod stress -p '{"spec" : { "containers" : [{"name" : "stress", "resources": {"requests": {"cpu":"100m","memory": "150M"}, "limits": {"cpu":"100m","memory":"150M"}}}]}}'

pod/stress patched

Looking good!

Our container stops restarting and is happily running now: