Six reasons your monorepo CI got slower this quarter

Originally published on got 95% of their monolith CI runs under 18 minutes, down from 45 — but only after it had silently grown the other direction for years. brought a "build all" job that had grown to take up to three hours down to around fifteen minutes.

These are after-photos. The before-photo is what every monorepo team eventually arrives at: median CI time creeps up, week by week, with no single bad commit to blame. Six mechanisms do the creeping. Here's what each one looks like, and which lever each one calls for.

You added packages

This one is mechanical. Your monorepo started with three packages. By quarter six it has fifty. Every step in your CI that runs across all packages — install, lint, typecheck, build, test — is now somewhere between five and sixteen times slower than it was at the start. Not because the work per package got harder. Just because there's more of it.

Gotcha: this is the easiest mechanism to spot, so it's the one teams blame first. It's almost never the headline cost.

Your slowest test got slower

The wall-clock time of any partially-parallel pipeline is bounded by the deepest single task in it. Your slowest end-to-end test, your largest integration suite, your one big database migration check — that single job sets a floor on how fast a green CI can possibly be, no matter how many runners you throw at the rest.

The critical path through your CI graph sets the floor. As your slowest job grows, that floor rises with it.

Shopify's CI team wrote in 2016 that "almost 16% of our tests have some level of flakiness associated with them" and that "about 84% of the transitions we observe from pass to fail involve a flaky test." That last number is the load-bearing one: when a green CI suddenly turns red, more than four times in five it's not a real bug.

The downstream cost is the retry. the task-level version. Their backend integration tests depended on a single set of storage containers — which meant "all tests were running against and overloading the single set of storage containers. Some tests required exclusive container access, leading to complex test parallelism and ordering." A test step "easily pushing over 50 minutes" wasn't slow because the work was hard; it was slow because the work couldn't run concurrently. The fix was hermetic tests — each with its own container — so they could finally run in parallel.

The other version lives a layer up: even when tasks are structurally parallelizable, the orchestrator running them may not see they're independent. Most build systems and CI providers treat their queue as a flat job list with hand-written dependencies — they can't see across packages, so independent work serializes into the order you wired it up.

Gotcha: when CI feels slow, the instinct is to look at the runners. The harder question is whether your work is even shaped to use them.

Setup tax

Setup overhead is bigger than people give it credit for.

Without a precomputed dependency graph, your build system has two options: rebuild everything, or rebuild based on a hand-maintained list that's always out of date. Most teams pick the first one, and pay for it on every PR — even the ones where most of the work was unnecessary.

The teams who've fixed this compute the impact set per change. Stripe runs only around 5% of their tests on average per change, against a 50 million line Ruby monorepo, because they identified which tests actually depend on the changed code.

Gotcha: the fan-out exists whether you measure it or not — but you can't skip what you haven't computed.

All six hit hardest when the cache is cold

Here's the thread that ties the six together.

You added packages, and on a cold cache every package gets re-installed and re-built from scratch. Your slowest test got slower, and on a cold cache you're paying the full duration of it instead of skipping it from cache. Your tests got flakier, and on a cold cache a retry isn't "just the test again" — it's the full chain again. Setup tax is at its worst right when the runner is brand-new, with no prefetched layers. Your graph fans out, and on a cold cache every node in the fan-out runs.

This is why the first PR of the morning takes 22 minutes — and the rest of the day is fine. The cache hides five of the six mechanisms once it's warm. The first PR pays the full bill.

Next week we'll look at what your cold-run dependency graph actually contains, using turbo run --dry=json — a tool you already have.

Vollständiger Original-Artikel

Den kompletten Beitrag mit allen Details direkt auf dev.to lesen.

↗ Original-Artikel auf dev.to lesen