Does the Model Really Need to Stream JSON?

Getting JSON out of LLMs has become fairly routine. You pass a JSON Schema, enable structured output, and if the model is capable enough, the final JSON is usually valid.

But what happens when you enable streaming?

A valid JSON document at the end of generation is not the same as having something your application can safely consume while it's still being generated.

A typical streamed JSON response looks something like this:

{"summary":"Export job failed...

Eventually this becomes a valid JSON object, but at this moment it isn't JSON at all.

In practice, this makes streaming JSON awkward to work with in real applications.

• A parse failure isn't really an error—it just means the response hasn't finished yet.


• It's difficult to tell which fields are actually complete.


• Even if the final output is valid, the intermediate state is broken almost the entire time.

This is simply a property of JSON as a document format. Until the document is complete, there is no stable intermediate object.

Does the model actually need to stream JSON?

Not necessarily.

What most applications want isn't for the model to gradually write out a JSON string. What they really need is structured state that remains usable throughout generation.

Instead of asking the model to stream a JSON document directly, I experimented with having it emit compact slot frames. Each slot ID maps to a JSON path, and incoming values are applied to a server-owned object state.

In other words, the model generates values, while the SDK assembles the JSON object.

For example, even if the model is still generating the summary field, the application can already receive:

{"summary":"Export job f"}

This isn't the final result, but it is a valid, serializable JSON object. It can be rendered immediately in the UI, sent over SSE, or passed to downstream services without special handling.

If a slot fails, only that slot needs to be regenerated. Fields that have already completed don't need to be generated again.

This is the approach behind slot-flight.

Benchmark

I compared two approaches using NVIDIA's openai/gpt-oss-20b endpoint.

• 
  slot-flight: the model streams slot frames, and slot-flight assembles the JSON state.


• 
  json-schema: uses NVIDIA's native response_format: { type: "json_schema" } streaming.

The results:

slot-flight

final schema ok: 40/40 = 100%

stream JSON stable: 100%

usable run: 40/40 = 100%



json-schema

final schema ok: 39/40 = 97.5%

stream JSON stable: ~2.0%

usable run: 39/40 = 97.5%

The native JSON Schema implementation produced reliable final results: 39 out of 40 responses passed schema validation.

However, only about 2% of streamed chunks represented a valid JSON state. For most of the generation, the client was looking at an incomplete JSON document.

With slot-flight, every intermediate state remained serializable as JSON, including slot-delta events.

A typical event looks like this:

{"type":"slot-delta","slot":"summary","value":"Export job f","state":{"summary":"Export job f"}}

Even though value is still being generated, state is already a complete object that the application can safely consume.

Conclusion

Users often want to see a summary before the entire response is finished. Servers may want to start downstream work as soon as individual fields become available. UIs naturally benefit from objects that fill in incrementally instead of appearing all at once.

In those situations, continuously trying to parse an incomplete JSON prefix isn't a great interface.

I built slot-flight not to answer the question, "Can an LLM produce valid JSON?"

The more interesting question is:

Can an application safely consume structured state before the final JSON document exists?

That's the problem slot-flight is designed to solve.

It also supports stable structured streaming for nested objects and arrays of objects.

https://github.com/miinhho/slot-flight