Why setTimeout is Lying to Your Retry Logic

You've written retry logic. It probably looks something like this:

async function withRetry(fn, retries = 3) {

  for (let i = 0; i < retries; i++) {

    try {

      return await fn();

    } catch (err) {

      if (i === retries - 1) throw err;

      await new Promise(r => setTimeout(r, 200 * (i + 1)));

    }

  }

}

You test it locally. You simulate a slow dependency like this:

const fakeDB = async () => {

  await new Promise(r => setTimeout(r, 200)); // simulate DB

  return { id: 1, name: 'test' };

};

Your retry logic works. Tests pass. You ship it.

Then in production, your app starts dropping requests under load.

The problem isn't your retry logic. It's your fake.

Real dependencies don't have flat latency

Here's what your Postgres instance actually looks like in production:

• 
  p50: 5ms — half of all queries finish in under 5ms


• 
  p95: 50ms — 95% finish under 50ms


• 
  p99: 200ms — 99% finish under 200ms


• 
  p99.9: 2000ms — that one unlucky query during a GC pause

Your setTimeout(fn, 200) simulates the worst case, every single time. That's not how production works. And because it's not how production works, your retry logic has never actually been tested against reality.

The bugs hide in the variance — not in the slow case, but in the unpredictability.

What the real distribution looks like

Latency in distributed systems follows a lognormal distribution. It's right-skewed: most requests are fast, a meaningful minority are slow, and a small tail is very slow.

This shape comes from how real systems work:

• 
  GC pauses — Java, Go, and even Node's garbage collector occasionally stops the world


• 
  Cold caches — first query after a cache miss is always slower


• 
  Network jitter — packet routing isn't deterministic


• 
  Noisy neighbors — other workloads on the same hardware compete for resources


• 
  Connection pool exhaustion — when all connections are busy, new queries wait

None of these are constant. They're random, rare, and multiplicative — which is exactly what produces a lognormal shape.

Why this matters for retry logic specifically

Consider this scenario: your p99 latency is 200ms and your timeout is 250ms.

With setTimeout(fn, 200), every test call takes exactly 200ms — safely under your timeout. Tests pass.

In production, the lognormal tail means 0.1% of calls take 500ms or more. Your 250ms timeout fires, your retry triggers, and now you're sending the same request again to an already-stressed database. Under load, this cascades.

This is the exact failure mode that causes retry storms — and it only appears in production because your local tests used flat delays.

The bugs that flat delays hide:

• Timeouts that are too tight for the real p99


• Retry logic that amplifies load instead of handling it gracefully
  


• Circuit breakers that never open during tests but open constantly in production


• Backoff strategies that feel correct locally but collapse under real variance

The fix: simulate real latency distributions

Instead of a flat delay, fit a lognormal distribution to real p50/p99 values and sample from it. Every call gets a different delay — most are fast, some are slow, a few are very slow. Just like production.

Here's the math:

function fitLognormal(p50, p99) {

  // p50 = median = e^mu  →  mu = ln(p50)

  // p99 = e^(mu + 2.326*sigma)

  const mu = Math.log(p50);

  const sigma = (Math.log(p99) - mu) / 2.326;

  return { mu, sigma };

}



function sampleLatency(p50, p99) {

  const { mu, sigma } = fitLognormal(p50, p99);

  // Box-Muller transform

  const u1 = Math.random(), u2 = Math.random();

  const z = Math.sqrt(-2 * Math.log(u1)) * Math.cos(2 * Math.PI * u2);

  return Math.exp(mu + sigma * z);

}

Call sampleLatency(5, 200) ten times and you'll get something like:

3ms, 7ms, 2ms, 12ms, 4ms, 180ms, 6ms, 3ms, 9ms, 440ms

That's what your database actually looks like.

Using slowdep

I built wraps any async function with zero dependencies and built-in presets for postgres, redis, stripe, openai, s3, and more

If your retry logic has never been tested against real latency variance, it probably has bugs you haven't found yet.

Source code and presets: github.com/arnnnavvvvv/slowdep