Switching to GPT-5.6 in production: it's a schema, cache, and harness problem — not a model swap

Ploy builds production marketing websites with an AI agent. They've been benchmarking every frontier release for months. Nothing beat Claude Opus until GPT-5.6 Sol — 2.2× faster, 27% cheaper, better visual scores.

Then they actually tried to ship it. That's where it got interesting.

"We use Vercel's AI SDK, but switching from Claude Opus 4.8 to GPT-5.6 Sol still exposed provider-specific assumptions throughout our stack."

What the numbers look like

GPT-5.6 also wrote significantly less code to get there — 2,508 characters of CSS against Opus's 17,957. Fewer tokens, less output, better result.

Fix your eval harness before you trust a single number

Before any of that was usable, Ploy had to fix their evaluation suite.

Tool-call budgets were sized for Opus's sequential style. GPT-5.6 makes parallel calls and exceeded those budgets on cases it was solving correctly. The eval executor didn't support batched file reads, which Opus rarely used and GPT-5.6 uses constantly. One dataset was missing an explicit minScore threshold, silently defaulting to 1.0 — so GPT-5.6 "failed" a hero image that scored 0.98.

About a third of raw failures in the first run came from harness assumptions, not model behaviour. Those failures were unevenly distributed in ways that would have made GPT-5.6 look worse than it was.

The lesson: if you're evaluating a challenger against an incumbent, triage the traces before trusting the pass rate. An eval that rewards the new model for behaving like the old one isn't measuring capability — it's measuring familiarity.

Tool schemas need rethinking per-provider

Ploy's code tool has 25 parameters. Claude sends the two or three it actually uses. GPT-5.6 sends all 25, filling unused slots with plausible-looking invented values (offset: 0, timeout: 120000, siteId: "00000000-...").

That sounds like a cosmetic annoyance. It wasn't. The tool implementation couldn't distinguish invented values from real ones, so 52–64% of file reads returned empty — but the tool still returned success: true. The model saw a success, read a blank result, compensated with more calls, and degraded.

Prompting didn't fix it. strict mode didn't fix it either (and would've required stripping schema validation). The working fix: a schema transform at the provider boundary that rewrites optional parameters as anyOf: [T, null]. GPT-5.6 sends explicit nulls for unused params, which get stripped before the tool runs. Empty reads dropped from 52% to 0%.

Prompt caching is a different API, not a different knob

Before fixing caching, GPT-5.6 appeared ~50% more expensive than Opus. After fixing it, it came out cheaper. The entire cost gap was cache misconfiguration.

The key difference: Anthropic caches at the organisation level — any conversation can hit a shared prefix entry. GPT-5.6 dropped implicit partial-prefix matching. Without explicit prompt_cache_key setup, a new conversation sharing Ploy's 29K static prefix cached 0% of it and paid the full uncached rate every time. GPT-5.6 also applies a 1.25× surcharge to every uncached prompt.

Ploy ended up with per-workspace cache keys and a layered breakpoint structure — static tools/prompt as one entry, workspace context as another, per-session context as a third. First-call hit rates went from ~0% to 83.7%. Total uncached input tokens fell 28%.

The third fix: reasoning replay

One more gotcha. GPT-5.6's Responses API replays prior-turn reasoning as server-side item references by default. Production conversations were intermittently failing with Item 'rs_...' not found. Setting store: false makes the SDK request encrypted reasoning content and replay self-contained blobs instead of server pointers. Worth checking if you're seeing intermittent context errors.

What to do

• 
  Evaluating GPT-5.6 against your current model? Audit your eval harness first — tool budgets, implicit thresholds, anything Opus-specific. Trust traces over aggregate pass rates.


• 
  Complex tool schemas? Check whether your implementation can handle fully-filled params. The nullable schema transform is the cleaner fix over prompting.


• 
  Running prompt caching? Test your actual cache hit rate at the API level. The providers cache differently — a config that works on Anthropic may hit 0% on OpenAI.


• 
  Using the Responses API with reasoning? Set store: false to avoid server-state pointer failures in long conversations.

The model benchmarks are real. So is the migration work. Plan for both.

Source: Ploy — Migrating a production AI agent to GPT-5.6

✏️ Drafted with KewBot (AI), edited and approved by Drew.