There are two main tools to help parameterise Kubernetes application deployments, Helm and Kustomize. While Helm focuses on allowing you to build or use others' applications based on parameterising manifests using templating, Kustomize avoids templating by allowing you to create overlays that can override a base configuration for specific deployments.
While the two can be used independently, they are more powerful when used together. The approach I use the most is to use Kustomize to declare your own custom app deployments and use Helm to deploy ready made deployments of 3rd party services.
However, I do run into a limitation of Kustomize frequently for larger, multi-app deployments, namely, repetition.
Frequently we have parameters that we would like to declare at a global level and inject them somehow into our various apps within our deployment. Think the hostname for a platform, which is shared by all of our app ingresses. If the hostname changes, we need to find and replace it in all of our ingresses. This can be accomplished with other Kustomize functionality (replacements), which will be illustrated further down, but as we will see, this can only take us so far.
Ultimately, the feature we are seeking is templating. Kustomize offers no templating option of its own, and the Kustomize maintainers have clearly stated that they do not plan to provide it. It is such a frequently requested feature that the Kustomize maintainers now require you to tick a box on feature requests stating that the feature you are requesting does not involve templating. This may because Kustomize has declared itself as the "template free configuration tool" for Kubernetes, which is fine. But with so many requesting the feature it leaves us with the question, how can we solve this problem on our own?
I would like to share the approach we have taken, as I think it would be of use. Firstly though, lets recap the built-in features that Kustomize offers to address duplication amongst manfiests. It may well be that your use case is already supported by Kustomize.
The Example Repo
I have created a repository to illustrate the problem and make the solution clearer. You can find it at , see "Delimiter" and "Index" sections. But this is about as far as replacements can take us. We cannot replace anything that isn't a Kubernetes resource field. To see an illustration of this point, read on.
Templating
We will move on to implementing templating in the worked example shortly. It is as simplified as possible, but demonstrates the point. First, to give more context, I will discuss a real world example.
Real World Problem
The original reason I had to implement this was AWS ACK Controller Role resources. They require you to define the permission policies in string json form. Here is an AWS IAM role for Cert Manager.
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
name: cert-manager
namespace: cert-manager
spec:
name: CertManager-dev
assumeRolePolicyDocument: >
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.eu-west-2.amazonaws.com/id/ABCDEGHIJKLMNOPQSTUVWXYZ"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.eu-west-2.amazonaws.com/id/ABCDEGHIJKLMNOPQSTUVWXYZ:sub": "system:serviceaccount:cert-manager:cert-manager"
}
}
}
]
}
inlinePolicies:
CertManager: >
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "route53:GetChange",
"Resource": "arn:aws:route53:::change/*"
},
{
"Effect": "Allow",
"Action": [
"route53:ChangeResourceRecordSets",
"route53:ListResourceRecordSets"
],
"Resource": [
"arn:aws:route53:::hostedzone/Z0123456789ABCDEFG"
]
}
]
}
We needed to change the cluster OIDC provider in the trust policy between our deployments, because each cluster has its own. There is no way to do this with Kustomize alone.
Worked Example
Back to our example, which is tag v0.1.3 in the repo. Let's say we want to modify the header shown in each app based on the environment we deploy to. This not only demonstrates that we can modify arbitrary parameters in our deployments, but that it doesn't even have to be a Kubernetes resource.
We will need to modify some of the text in the files/index.html file in each app.
I was looking for a general Go templating style CLI that I could use as part of my build command. In (Gomplate)[] and minimal configuration required. That being said, there is some recommended configuration that I suggest, which I will cover.
You can see the syntax that Gomplate uses in their (docs)[ has been updated.
This has been a simple example, but hopefully you can see how we can extend this by using multiple config.yaml files for different deployments. When combined with the app specific configuration options that Kustomize provides, we have the ability to parameterise anything we want in our deployments.
Summary
To implemement deployment global parameters we can use Kustomize replacements to a point. If you require more complex templating features, you can implement Gomplate as part of your manifest build pipeline. The suggested implementation should not interfere with any existing Kustomize (or Helm, if you also use it) builds.
SOCIAL SHARE CARD GENERATOR