This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.
There's a special kind of bug that a retry loop can't save you from: the request that never fails. It doesn't throw. It doesn't time out. It just… sits there, socket open, forever — and your carefully-written exponential backoff waits right alongside it, because backoff only runs when something errors, and nothing did.
I hit exactly that while running a large batch of text-to-speech jobs through Google's Gemini TTS to narrate long-form content. Here's how a job that was "still running" for hours turned out to be doing nothing at all, and the one-line reframe that fixed it: a hang is not an error, so give every await a deadline.
The setup
The pipeline is simple in shape: take a long piece of text, split it into paragraph-sized chunks, send each chunk to Gemini TTS, get back audio, then concatenate the clips into one file. Dozens to hundreds of chunks per run. Because any single network call can flake, each chunk was wrapped in retry-with-backoff:
async function ttsWithRetry(chunk) {
for (let attempt = 0; attempt < 4; attempt++) {
try {
return await callGeminiTTS(chunk); // <- the await that betrayed me
} catch (err) {
await sleep(2 ** attempt * 1000); // 1s, 2s, 4s, 8s
}
}
throw new Error("TTS failed after retries");
}
Clean. Resilient. Ran beautifully for small batches.
The symptom
On the big runs, the job would chew through most of the chunks, gradually slow down, and then simply stop making progress. No error. No crash. No log line. The process was still alive — it just never produced another clip.
The tell was in the system stats: the process sat at 0% CPU with no new output files appearing. A crashed job is dead; this one was a ghost — present, "running," accomplishing nothing. And my backoff, the thing I'd added specifically to survive network trouble, never printed a single retry.
Why the retries never fired
I checked the connections and there it was: an ESTABLISHED but completely silent socket. The request had gone out, the connection was open, and then… nothing came back. Not an error, not a reset, not a close. Just an open pipe with no bytes flowing.
That's the whole bug in one sentence: my retry logic only handled failures, and a hang is not a failure. callGeminiTTS(chunk) never rejected, so the catch never ran, so backoff never fired. await will wait as long as you let it — and I had told it to wait forever. Under sustained load, an occasional connection would degrade into this silent state, and one stuck chunk was enough to freeze the entire batch behind it.
The fix: race every call against a deadline
The repair wasn't a bigger retry count or a fancier backoff curve. It was to make a hang look like the error it morally is — by racing every TTS call against a timeout, so a stall becomes a rejection that backoff can actually catch:
function withTimeout(promise, ms, label) {
return Promise.race([
promise,
new Promise((_, reject) =>
setTimeout(() => reject(new Error(`timeout after ${ms}ms: ${label}`)), ms)
),
]);
}
async function ttsWithRetry(chunk) {
for (let attempt = 0; attempt < 4; attempt++) {
try {
// a silent socket now loses the race and REJECTS in 180s
return await withTimeout(callGeminiTTS(chunk), 180_000, chunk.id);
} catch (err) {
await sleep(2 ** attempt * 1000);
}
}
throw new Error("TTS failed after retries");
}
Now a stuck request rejects after 180 seconds, the catch runs, backoff kicks in, and the next attempt opens a fresh connection instead of inheriting the dead one. The batch stopped freezing. Runs that used to wedge indefinitely now self-heal and finish.
Two things carried the win:
Every network await got an upper bound. Anawaitwithout a timeout is an unbounded promise to wait — and "forever" is a value your program can actually take. Bounding it turned an un-catchable hang into an ordinary, retryable error.
Detection got a real signal. I stopped assuming "process alive" meant "work happening." The reliable liveness check was new output files over time + CPU, not the mere existence of the process.
Before / after
Before: one degraded connection to Gemini TTS silently stalls a chunk → the whole batch hangs at 0% CPU → retries never fire because nothing errored → I babysit and kill it by hand.
After: a stall loses a 180s race → rejects → backoff opens a new connection → batch finishes on its own.
Where Google AI comes in
The engine doing the actual work here is Google's Gemini TTS, turning text into natural narration at batch scale. Google AI made the product possible; this fix made the pipeline around it reliable — which is the unglamorous half of shipping anything on top of a model. The most valuable thing I did for a Google-AI feature this quarter wasn't a prompt. It was giving its network calls a deadline.
The lesson
Retry logic guards against failure. It does nothing against silence. If a call can hang — and any network call can — then "resilient" means every single await has a deadline, and a stall is converted into an error your existing recovery can see.
A hang is not an error. So make it one. This bug is smashed — the batch now fails fast and heals itself instead of waiting politely, forever, for a socket that already gave up.