Streaming Article Queue Progress Over SSE

The article queue showed one opaque Generating spinner while three separate drafts were being researched, generated, checked, and saved.

That was technically accurate and practically useless. The server knew much more than the interface admitted, so I replaced the spinner with a live three slot stepper driven by server sent events.

Streaming the actual work

POST /api/articles/drafts/queue now streams progress over SSE. Each event reports meaningful state for the current slot: Researching, Generating, Verifying links, or Done.

The dashboard renders those events through GenerationProgress. Instead of staring at a spinner and guessing whether anything is happening, I can see source counts appear during research, draft titles arrive after generation, and completion land only when the draft actually exists.

That last detail matters. slot_done carries the real draft_id, and it fires at honest timing. A cheerful Done state before persistence finishes is not progress reporting. It is fiction with a loading indicator.

Progress had to reach the real work

Adding SSE at the route was the easy part. Useful progress had to originate deeper in the generation path.

I threaded progress_cb through backfill and generate_article_payload, where the actual stages happen. That keeps the events tied to work being performed instead of having the route estimate progress with timers or decorative percentages.

There are no fake claims like 73 percent complete. Article generation does not provide a clean linear unit of work. Named stages and concrete artifacts are more honest: sources found, title generated, links verified, draft persisted.

Failure should preserve completed work

Queue fills now insert drafts one slot at a time. If slot three fails, slots one and two remain persisted and visible.

Previously, treating the queue fill as one indivisible operation would make the interface simpler, but it would also discard useful completed work or hide it behind the later failure. Per slot persistence gives partial success first class status.

Errors stay inside the relevant stepper slot. The user sees which draft failed without losing the progress and results from the other slots. That is a much better failure model than collapsing the entire queue into one red banner.

The compatibility tradeoff

SSE makes the browser experience substantially better, but not every client expects a stream. I kept the JSON fallback for clients that do not consume SSE.

That creates two response paths to maintain. It is extra surface area, but forcing every existing client to adopt streaming would turn a dashboard improvement into an unnecessary compatibility break. The tradeoff is worth it while both clients exist.

What I would do differently

I would define the progress event schema before threading callbacks through the stack. The stages, payload fields, error shape, and completion semantics are the real contract. Starting there would make the backend and dashboard converge faster, especially around the precise moment a slot becomes Done.

The broader lesson is simple: if the server knows what is happening, the interface should not pretend it only knows that something is happening.