Introduction


We had an easy-sounding feature: a realtime assistant that streams model responses to users over WebSockets. It worked in dev, and even in staging.

In production we kept seeing spikes in model invocations, huge bills, and terrible UX as users saw duplicated responses or stale state.

This is what we learned the hard way.




...