How Systematic EBS Optimisation Delivered £108,070/Year (Without Breaking Production)

EBS volumes don’t fail loudly—they fail quietly on your bill.

In one production EKS cluster supporting 1000+ namespaces, I found 1,289 unattached EBS volumes. Many looked safe to delete, but “unattached” isn’t the same as “orphaned”—and fast cleanups are often the ones that cause incidents.

So instead of deleting aggressively, I ran a controlled verification campaign with phased execution and team validation.

Results (organisational impact):

• 
  732 volumes removed (50.9TB)


• 
  £108,070/year saved (≈ £9k/month)


• Zero production incidents


• Insights carried over to another cluster where 3 high‑IOPS volumes delivered £45,241/year on their own (≈ £15k/volume)

Public evidence (ticket thread + outcomes):

• https://github.com/ministryofjustice/cloud-platform/issues/7662

Generalised methodology (public):

• 
  https://github.com/olu-folarin/aws-ebs-optimization-methodology-
  This is a sanitised, organisation‑agnostic version of the framework. The implementation scripts used at work remain internal.
  

Definitions & scope

Orphaned (in this context): a volume that is unattached, not referenced by PV/PVC, not part of backup/DR infrastructure, not depended on by active restore/image workflows, and validated safe to delete where uncertainty exists.

Scope: production EKS cluster (live-1) in eu‑west‑2, supporting 1000+ namespaces, executed under change control with approvals and audit‑friendly evidence.

Timeline (shows the pace)

• 
  21 Oct 2025: ticket assigned + investigation started


• 
  6 Nov 2025: medium‑value deletion work completed


• 
  10 Nov 2025: team validation session for the final high‑value candidates; additional high‑IOPS volumes removed in another cluster

The verification methodology

This is the refined, reusable version of the approach I recommend (the execution itself was iterative—more on that below).

Step 1: AWS attachment status

Confirm the volume is actually unattached (candidate, not conclusion).

aws ec2 describe-volumes --filters Name=status,Values=available

Step 2: Age + cost signal (especially io1/io2)

Prioritise what to review first by:

• days unattached


• volume type


• IOPS provisioning (io1/io2 can hide the real cost)

The £15K discovery: a 750GB io1 volume with 19,500 provisioned IOPS.

That single volume represented a meaningful chunk of the savings. The takeaway is simple: if you don’t check IOPS, you can miss the real spend.

Step 3: PersistentVolume (PV) reference check

PV answers: does the cluster still have an infrastructure binding to this volume?

kubectl get pv -A -o wide | grep vol-

If a PV exists (even in Released), treat it as a “pause and investigate” signal.

Step 4: PersistentVolumeClaim (PVC) check

PVC answers: does any workload still claim this storage?

kubectl get pvc --all-namespaces

If a PVC exists, assume an application still expects it—regardless of attachment state.

Step 5: Backup / DR signals (Velero, etc.)

Volumes that look “orphaned” can still be disaster‑recovery critical. Tagging/labels and documented recovery workflows matter here.

Step 6: Snapshot / AMI / restore pipeline dependencies

A volume doesn’t “back an AMI” in the way people often think—snapshots and images can outlive the source volume. The real question is:

Is this volume part of an active restore/image pipeline (referenced by automation, launch templates, image build processes, or documented recovery workflows)?

If yes, preserve it until ownership and lifecycle are clear.

Step 7: Team validation for high‑value candidates

For the final, highest‑impact volumes, I ran a validation session with senior engineers:

• I walked through the evidence for each candidate


• they independently verified using their own checks


• we deleted during the session with shared confidence and accountability

This step matters because high‑impact changes shouldn’t rely on a single person’s judgement—even when the analysis is strong.

Phased execution

The campaign was executed in phases with monitoring between batches. The principle was:

Delete safely at scale, not quickly in one go.

For high‑risk candidates, a good safety net (when appropriate in your environment) is a snapshot + tested recovery path (restore snapshot → create volume → reattach → confirm workload) before deletion.


In regulated environments, the “prove you can recover” step is often as valuable as the deletion itself.

Results (direct vs organisational impact)

Direct campaign (my execution):

• 729 volumes removed


• 50.9TB reclaimed


• £62,829/year saved


• Zero production incidents

Additional impact (after the campaign):

• In another cluster, 3 additional io1 volumes showing the same “IOPS dominates cost” pattern were removed


• 
  £45,241/year saved from those 3 volumes alone (≈ £15k/volume)

Combined organisational impact: £108,070/year from 732 volumes.

Common failure modes this method avoids

• 
  Released PVs that still matter: PV lifecycle doesn’t always match application expectation.


• 
  Backup/DR volumes that look idle: “No PV/PVC” doesn’t automatically mean “safe”.


• 
  io1/io2 where IOPS is the bill: storage is visible; IOPS often isn’t.

Why the adoption signal matters more than the headline number

£62,829 validated the approach.

But the real leverage was that the same checks highlighted a repeatable cost pattern (high‑IOPS volumes provisioned for workloads that no longer needed that tier), leading to additional savings beyond the original cluster.

That’s the difference between:

• a one‑off cleanup, and


• a reusable framework that changes how teams evaluate “orphaned”.

A note on methodology (intellectual honesty)

The public ticket documents the work as it happened. The 7‑step framework above is the refined version—informed by what I learned during execution (especially the IOPS discovery). Real engineering is iterative; this is what I recommend you use going forward.

What’s missing: prevention

Cleanups buy you time, but they don’t fix the root cause. Prevention looks like:

• tagging volumes at creation (owner, service, environment)


• automated detection: “unattached > 30 days”


• cost visibility dashboards (EBS spend by team/namespace)


• enforcement at merge time (policy/guardrails)

Resources

• Public evidence (ticket): https://github.com/ministryofjustice/cloud-platform/issues/7662
  


• Generalised methodology (public): https://github.com/olu-folarin/aws-ebs-optimization-methodology-
  


• AWS EBS pricing: https://aws.amazon.com/ebs/pricing/
  


• FinOps Foundation: https://www.finops.org/
  

Have you run cost reduction campaigns that had to be both safe and repeatable? What checks helped you avoid incidents?