The debug copy of our service was taking one request in three

A colleague opened our staging admin tool to check something unrelated and found a real customer's order in it, placed forty minutes earlier, with a real card reference on it. For nineteen days about a third of our checkout writes had gone into the staging database. Three weeks before, an engineer had needed to reproduce a bug against staging...