Exponential backoff with jitter stopped our CI retry storms

TL;DR: Exponential backoff with jitter spreads client retries over time so a recovering service doesn't get flattened by a synchronised wave. We added full jitter to our CI agents and a retry budget, and our internal 503 rate during recovery dropped from minutes of saturation to seconds.

A few months back at Buildkite a metadata service we run on ECS had a 40 second blip, and the recovery took six minutes instead of forty seconds. The service came back healthy, then immediately fell over again, then recovered, then fell over. Our build agents had all retried at the same fixed interval, so every two seconds a few thousand requests arrived in lockstep. That synchronised wave is a retry storm, and exponential backoff with jitter is the standard fix.

I'd read about this for years and never actually watched it bite us. Once it did, the maths got a lot more interesting. Here's what we changed and why it worked.

What is a retry storm

A retry storm happens when many clients fail at the same moment, then retry on the same schedule, producing a repeating spike of traffic that prevents the downstream service from recovering. Because the retries are correlated in time, each recovery attempt is immediately overwhelmed, so a short outage stretches into a long one. Jitter breaks the correlation by randomising when each client retries.

The key word is correlated. One client retrying every two seconds is nothing. Three thousand clients that all failed at 12:04:01 and all retry at exactly +2s, +4s, +8s is a denial-of-service attack you wrote yourself. The downstream never sees the gaps between the spikes, so its CPU and connection pool stay pinned.

Why fixed-interval retries make it worse

Our old agent retry config looked sensible on paper. Retry up to 5 times, wait 2 seconds between attempts, give up after that. The problem is the 2 seconds is identical for every agent, so the herd stays in formation through the entire recovery window.

Plain exponential backoff helps a bit. If you double the wait each time (2s, 4s, 8s, 16s) the spikes spread out as attempts progress. But the first retry after a shared failure still lands at the same instant for everyone, and that first wave is usually the one that re-saturates a fragile service. The fix that actually matters is adding randomness on top of the exponential growth.

describes this pattern, and it's the single change that put a hard ceiling on storm amplitude.

Second, a circuit breaker in front of the client. After a threshold of consecutive failures it opens for a few seconds and returns errors locally without touching the network. That gives the downstream a guaranteed quiet window to recover, instead of a steady trickle of probes from thousands of agents.

Trade-offs and limitations

Jitter makes individual request latency less predictable, which annoys anyone trying to set a tight p99 SLO on a single call. You're trading per-request determinism for system-wide stability, and that's usually the right trade for internal infrastructure, but not always for a user-facing path with a strict deadline.

Retry budgets need per-client state, so a brand new agent with no history can still pile on during the first window. Circuit breakers add a threshold and a timeout you have to tune, and a breaker set too aggressively will trip on normal noise and hurt availability. We game-day these settings now rather than trusting the defaults, because a control you've never tested under failure is just a guess.

Wrapping up

Exponential backoff with jitter is cheap to add and it solved a problem that fixed-interval retries had quietly created for us. If I were starting over I'd reach for full jitter by default, pair it with a retry budget, and only add a circuit breaker once I'd measured a real cascading failure. Then I'd run a game day to break it on purpose, because the only retry logic I trust is the kind I've watched fail safely.

Further reading