, all due to a crash that was not only unpredictable, but entirely avoidable. And it could happen to you too.
What happened
I have Postgres running as a Kubernetes deployment. In my values.yaml, I had set the following:
postgresql:
image:
tag: 15
It worked perfectly for months until this morning, when I woke up to my database in a crashloop and my service completely unavailable. Why? It turns out K8s had to restart the PostgreSQL pod, which then loaded the latest 15.* version of the Docker image. Unfortunately for me, that image is, I suggest you read up on it. In a nutshell, it is a standard/guideline for defining version numbers and how to increase them:
Given a version number MAJOR.MINOR.PATCH, increment the:
MAJORversion when you make incompatible API changes
MINORversion when you add functionality in a backward compatible manner
PATCHversion when you make backward compatible bug fixes
Additional labels for pre-release and build metadata are available as extensions to theMAJOR.MINOR.PATCHformat.
"Pinning" and why it matters
In my case, by setting the Docker image version to 15, I gave Kubernetes the freedom to download any minor or patch version of it - at any time, without warning. By "pinning" the version - in other words specifying MAJOR, MINOR and PATCH version explicitly - I regained control and the ability to update the image version on my own terms and test it thoroughly in my setup.
To summarize, this is bad:
postgresql:
image:
# Will potentially crash your application at any point
tag: 15
Stay safe by only allowing the latest PATCH version:
postgresql:
image:
# This will pull the latest 15.7.x patch version
tag: 15.7
Final thoughts
Even with 15 years of experience, rookie mistakes creep in. Not pinning your dependencies is an easy mistake to make, but can have dramatic consequences for your product and your company.
Ironically, the product I was about to launch could have actually helped me catch this issue before merging it into production 🤦♂️ Check out PR Manager, where you can generate reviews for your code changes before merging them. It allows you to catch issues like the one in this post and lets you fix them with the click of a button.
SOCIAL SHARE CARD GENERATOR