TL;DR — Constrained decoding and JSON schema enforcement guarantee that model output parses — they say nothing about whether the values are true, safe, or grounded in real system state. Treat structured output like you'd treat any untrusted client at an API boundary: schema validation is step one, not the whole job. The dangerous bugs left in production aren't malformed JSON anymore; they're well-formed lies.
Structured output was sold as a solved problem. Constrain the decoder to a grammar, force the model to emit valid JSON against a schema, and the parsing errors that plagued early LLM integrations disappear. They did disappear. But somewhere along the way, teams started treating schema-valid as a synonym for correct, and that substitution is quietly causing a new class of production bugs that don't look like bugs at all — they look like clean, well-typed data.
This is worth being precise about, because the word "type" is doing a lot of unearned work in how people talk about function calling and structured generation.
The Guarantee You Actually Got
Grammar-constrained decoding gives you exactly one guarantee: the output will conform to a shape. If your schema says status is an enum of three strings, the model will emit one of those three strings. If it says amount is a number, you get a number, not a sentence fragment. That's real, and it's valuable — it eliminated an entire category of glue code that used to exist purely to recover from malformed responses.
But a type system in the compiler sense does more than check shape. It checks that a value belongs to a domain that makes the rest of your program sound. A JSON schema can say user_id is a string. It cannot say user_id refers to a user that exists. It can say refund_amount is a positive number. It cannot say that number is less than or equal to the actual balance on the account. It can say date matches an ISO format. It cannot say the date isn't in the future for a field that logically can't be in the future. Grammar constraints operate entirely inside the syntax of the value. They have zero visibility into the semantics of your domain, because that semantics lives in your database, your business rules, and the current state of the world — none of which the decoder has access to at generation time.
Well-Formed Lies
Before structured output, a bad model response usually broke your parser. You got an exception, a retry, a visible failure. That failure was annoying but it was honest — the system told you something was wrong.
Now the failure mode has changed shape. The model still hallucinates, still misreads context, still guesses under uncertainty — but it does all of that inside a perfectly valid JSON object. The enum value is real, just the wrong one. The order ID is correctly formatted, just doesn't exist. The function call has the right argument types, just the wrong argument values. Nothing throws. Nothing logs an error. The malformed response of two years ago has become the well-formed lie of today, and well-formed lies are much harder to catch because your existing observability was built to catch parse failures, not semantic ones.
This is the same mistake web developers made with client-side form validation fifteen years ago: confusing "the browser won't let the user submit garbage" with "the server can trust what it receives." We relearned that lesson the hard way once already. Structured LLM output is asking us to relearn it a second time, in a domain where the "client" is a probabilistic model instead of a browser, which makes the untrusted input even less predictable, not more.
Function Calling Is an RPC Client You Didn't Write
Function calling makes this sharper because the stakes go up. A structured object that's wrong sits in a database field. A function call that's wrong executes. The model is choosing which method to invoke and with what arguments, and the tool schema you gave it is documentation, not a contract in the enforceable sense. The schema can constrain the type of an argument. It cannot encode the precondition that a refund can't exceed the original charge, that a cancellation can't target an order that already shipped, that a permission-scoped action can't be invoked for a resource outside the caller's tenant.
Those preconditions exist in your codebase already — they're the same invariants you'd check for any caller, human or programmatic. The mistake teams make is skipping that check specifically for LLM-originated calls, on the theory that schema compliance already did the validating. It didn't. The LLM is best modeled as an RPC client you didn't write, running code you didn't review, calling into your system with arguments generated by a process that has no concept of your business rules unless you put them there explicitly and check them again downstream.
Two Validation Layers, Not One
The fix is unglamorous and it's the same fix that's always existed for untrusted input: layer your validation instead of collapsing it into one step.
Schema validation at the boundary — shape, type, required fields. This is what constrained decoding and JSON schema already give you for free. Keep it, don't over-invest in it further.
Semantic validation after parsing, before any side effect. Does this ID resolve to a real, accessible entity? Is this value within the domain's actual legal range, not just its syntactic range? Does this combination of fields represent a state your system can actually be in? This layer has to be hand-written, because it encodes knowledge the model's grammar cannot see.
Neither layer replaces the other. Teams that only run the first layer are shipping the equivalent of an API that trusts its own request validator to also be a database consistency check. Teams that run both are treating LLM output the way they'd treat any input from a system they don't fully control — which, given what an LLM actually is, is exactly the right level of trust.
Measuring the Gap
If you evaluate your structured-output or tool-calling pipeline only on schema compliance rate, you are measuring the layer that was already guaranteed by construction. It will look great and tell you almost nothing about production risk. The metric that matters is the gap between schema-valid and semantically-valid — how often does a well-formed response fail your domain checks after it parses cleanly? That number is your actual hallucination rate for structured tasks, and it's usually far more informative than any aggregate accuracy score, because it isolates exactly the failure mode your syntax guarantees were never designed to catch.
Structured output didn't make LLM integrations safe. It made them legible. Legibility is genuinely useful — you can't validate what you can't parse — but it's the beginning of the trust boundary, not the end of it. Treat the schema as a cast, not a proof, and build the semantic checks you'd build for any other untrusted caller. The model will keep being confidently wrong inside perfectly valid JSON. Your validation layer is the only thing standing between that confidence and your production data.
SOCIAL SHARE CARD GENERATOR