A staging-to-production data bleed cost me 4 hours of rollback. That's what finally made dry-run a structural requirement, not an afterthought.
The common advice is: test in staging, promote when green. The problem is environment drift. My D1 schema changes once or twice a week, and a solo operator can't keep staging perfectly synchronized. Worse, agents don't have fixed execution paths — the same input can produce a different tool call sequence on the next run. I ran a flow 100 times in staging and still hit a fresh path on the first production execution.
The most surprising thing I learned after 6 months of running this: latency wasn't the problem I expected. KV writes averaged 12ms — basically imperceptible. The real problem was that mock responses fool the agent into treating skipped writes as real successes. I'd dry-run an R2 put, the agent would believe the file was uploaded, and then proceed to write metadata to D1 — which was not in dry-run scope. Real write, orphaned record.
The fix: once any write tool in a run hits dry-run, propagate a flag for that runId that forces all subsequent writes in the same run to dry-run too.
// after intercepting first dry-run write
await ctx.env.KV.put(`dryrun_active:${ctx.runId}`, "1", {
expirationTtl: 3600,
});
// every subsequent hook checks this flag
const isDryRunActive =
(await ctx.env.KV.get(`dryrun_active:${ctx.runId}`)) === "1";
One more thing that burned me: if the hook itself fails — say, KV goes temporarily unavailable — Claude Code's default behavior is fall-through. The tool call executes anyway, dry-run flag ignored. Last week a KV spike caused hook timeouts and 3 agents wrote directly to production. No data loss because those ops were idempotent, but it was luck. Hook failure needs its own alert, separate from agent failure.
I wrote up the full breakdown — including the dry-run propagation edge cases, R2 + D1 orphan scenarios, and where this pattern completely falls apart (read-modify-write loops, APIs with side-effectful reads) — over on riversealab.com.