Jenkins as a Code, or how I stopped clicking around in the UI

I've been running Jenkins in one form or another for years now. Different companies, different sizes of teams, but somehow the same story keeps repeating itself, and at some point I just couldn't take it anymore. So I decided to write down what I went through, what I learned, and where this journey took me. This is Part 1 of what I'm calling My CI/CD Odyssey — a series where I want to share the ideas, the mistakes, and the things that actually worked.

Future chapters will go deeper into the painful stuff — building macOS workers without losing your mind, using spot instances as GitHub Actions runners to cut costs, and a few other rabbit holes I went into. But before we get there, let's start at the beginning, because the beginning is where most of the pain lives.

The "before" picture, and why it hurts

If you've worked with Jenkins for any reasonable amount of time, you probably know this scene: someone opens the Jenkins UI, clicks "New Item", picks a freestyle or pipeline job, fills in twenty-something fields, scrolls past a wall of plugin options, and clicks Save. Then a month later somebody has to figure out why a job behaves differently in dev than in prod, and the answer is "because Arthur clicked a different checkbox in February and nobody remembers".

That was basically my world for a long time. We had multi-tier environments — dev, stage, sometimes more — and on top of that, sometimes more than one Jenkins instance per tier. Each one was configured by hand. Plugins installed by hand. Pipelines copy-pasted from one Jenkins to another and edited by hand. Credentials added by hand. Workers attached by hand. Then one day you wake up and realize:

• Nobody remembers what plugins are installed where.


• The "stage" Jenkins doesn't match production anymore, and you only notice when a pipeline breaks in prod.


• A plugin update on Friday afternoon kills a build, and rolling it back means a human clicking buttons under stress.


• A new team member joins and you spend three days explaining tribal knowledge that should really live in a repo.

That last point is what really got me. Tribal knowledge is fine when there are two of you. It stops being fine very quickly.

The idea: treat Jenkins like any other piece of code

So I started doing some research, and the direction was pretty obvious in hindsight: if Jenkins is a piece of infrastructure, and we treat infrastructure as code everywhere else (Terraform for cloud, Helm for Kubernetes, Ansible for hosts), then Jenkins itself shouldn't be the special snowflake we manage by hand. The whole controller, all the jobs, all the credentials wiring, the workers — everything should come out of a git repo. End to end.

The goal I wrote down for myself was something like this:

I want a Jenkins instance where I can throw away the whole VM, the whole cluster, the whole config, run a pipeline, and ten minutes later have an identical Jenkins back. And I want dev to be code-to-code identical to prod, so when I test a plugin upgrade or a pipeline change in dev, I actually know it will behave the same in prod.

If you've ever burned yourself on a "but it worked in stage" deploy, you know exactly why that sentence matters.

The building blocks

Once I started designing this, the picture broke down into a few moving pieces. None of these are revolutionary on their own — what matters is how they fit together.

is a Jenkins plugin that lets you define the entire controller config in YAML. System settings, security realm, authorization strategy, clouds, credentials wiring, tools, global libraries — all of it. The controller reads the YAML on boot and configures itself.

The moment I plugged JCasC in and could rebuild a controller from a YAML file, I knew I wasn't going back. No more "what's installed where". Whatever is in the YAML is the truth. If it's not in the YAML, it doesn't exist.

A minimal taste of what that looks like:

jenkins:

  systemMessage: "Managed by JCasC — do not edit in the UI"

  numExecutors: 0

  mode: EXCLUSIVE

  securityRealm:

    github:

      clientID: ${GITHUB_CLIENT_ID}

      clientSecret: ${GITHUB_CLIENT_SECRET}

  clouds:

    - kubernetes:

        name: "eks"

        namespace: "jenkins"

        jenkinsUrl: "http://jenkins.jenkins.svc.cluster.local:8080"

unclassified:

  globalLibraries:

    libraries:

      - name: "ci-libs"

        defaultVersion: "main"

        retriever:

          modernSCM:

            scm:

              git:

                remote: "https://github.com/<org>/ci-libs.git"

Fifteen lines, and the whole controller knows who it is.

2. Job DSL — jobs from a git repo

JCasC handles the controller, but it doesn't really handle jobs. For that I leaned on the

3. Helm + Kubernetes for the controller

I run the Jenkins controller in , persistent volume for the home dir, a sidecar that injects JCasC config from a ConfigMap. Upgrading Jenkins is just bumping a chart version. Rolling back is rolling back a chart version. Plugin lists are values in a Helm values.yaml file, version-pinned, and reviewed in a pull request like any other change.

This is honestly the part that made plugin upgrades stop being scary. They go through a PR. They get tested in dev first. They get the same review as application code.

Side note: if you'd rather not deal with Helm at all, the community also maintains a . Every worker image is baked from a Packer template that lives in git: base OS, language runtimes, SDKs, build tools, everything pre-installed. The image gets a version. The version gets pinned in the worker config.

This was the moment that builds started to feel reproducible. Before Packer, every worker was a slightly different snowflake, hand-installed and slowly drifting. After Packer, every worker that boots from image v1.2.3 is byte-for-byte the same as every other worker booted from image v1.2.3. If a dependency upgrade breaks something, you know exactly which image introduced it, and you can pin back to the previous one in a one-line PR.

5. Ephemeral workers — born, used, destroyed

This is the part that connects everything, and honestly the part I'm proudest of. Workers in this setup are ephemeral. Not "long-lived agents we reboot once a week" — actually ephemeral. A pipeline asks Jenkins for a worker, dedicated job spins one up from a known Packer image, the worker runs the build, the worker dies. Always. Every build gets a virgin environment.

The "something" depends on the platform, but the pattern is identical across all of them:

• 
  Linux builds — the to provision and de-provision instances from packer templates.


• 
  macOS VMs — same idea, but the underlying virtualization is its own world. We spin up a fresh macOS VM from a Packer-baked image on each build (via Tart on Apple Silicon hosts, or vSphere for older fleets, or
  
  
  

  
  
  
  6. Terraform / Terragrunt for everything else
  

  
  
  

  All the things that aren't Jenkins itself — VPCs, IAM, secret stores, the EKS cluster, image galleries — live in so the same modules get reused across dev and prod with different inputs. Same code, different variables. That's how I get dev to be code-to-code identical to prod.

  
  
  

  If you ever want to test how production will behave, just run the same Terraform with ENV=stage instead of ENV=prod. Same modules, same versions, just a different namespace. No surprises.

  
  
  

  
  
  
  How it all clicks together
  

  
  
  

  The flow ends up looking like this:

  
  
  
  
  
  1. Somebody opens a pull request — could be a new job, a plugin bump, a JCasC tweak, a new Packer image.
  
  
  2. CI runs validation: YAML lint, Groovy compile checks, Terraform plan, Packer build for changed images.
  
  
  3. PR gets reviewed and merged.
  
  
  4. On merge,
     
     
     
     
     
     • 
       No more "works on stage, breaks on prod". Because the two are literally the same code with different inputs. If it works on stage, it works on prod, modulo data differences.
     
     
     • 
       Plugin upgrades stopped being scary. They go through a PR. They get tried on dev. They roll back with git revert.
     
     
     • 
       Onboarding got faster. New engineers read the repo. They don't have to be told secrets or shown a Jenkins UI tour.
     
     
     • 
       Disaster recovery got real. I can lose the controller VM, the EKS cluster, even the entire account, and as long as I have the repo I can rebuild.
     
     
     • 
       Audit trail came for free. Every change to any pipeline is a git commit, with an author, a timestamp, and a PR description. No more "who changed this and when".
     
     
     
     
     

     
     
     
     What I'm still figuring out
     

     
     
     

     I don't want to make this sound like a finished story, because it's not. A few things still keep me up at night:

     
     
     
     
     

     • macOS workers are their own special kind of hell. You can't just spin up a Mac VM in AWS the same way you spin up Linux. There's a whole ecosystem of hypervisors, licensing rules, and hardware constraints to deal with. This deserves its own post — and it's getting one. Part 2 will be all about macOS workers: — the controller config in YAML.

     
     
     • 
       — ephemeral pod agents in EKS.
     
     
     • 
       — what I use to deploy the controller.
     
     
     • 
       — bakes all the worker images (Linux, Windows, macOS).
     
     
     
     
     

     Infrastructure

     
     
     
     
     
     • 
       — keeps the same modules DRY across dev / stage / prod.
     
     
     • 
       — where the Jenkins controller lives.
     
     
     • 
       — applies Terraform on merge.
     
     
     
     
     

     Coming up in later parts

     
     
     
     
     
     • 
       — Tart cluster orchestration for macOS fleets (Part 2).
     
     
     
     
     
     
     
     

     This is Part 1 of My CI/CD Odyssey. If you want to be pinged when Part 2 drops, follow me here on dev.to. And if you're doing JaaC differently — I'd love to hear about it in the comments.

     Vollständiger Original-Artikel

     Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.

     ↗ Original-Artikel auf dev.to lesen