How a model upgrade silently broke our extraction prompt (and how we caught it)

A friend's product summarizes customer support tickets using a fine-tuned LLM

prompt. It worked perfectly on GPT-4o for six months. Then OpenAI deprecated

4o, the team migrated to GPT-4.1, ran a smoke test in the playground, said

"looks fine," and shipped.

Two weeks later a customer escalated: "Your urgency tagging is wrong on

basically everything since last Wednesday."

The prompt asked for {"intent": "...", "urgency": "low|medium|high"}. On

4o, the model returned exactly that. On 4.1, it started returning

{"intent": "...", "urgency_level": "..."} — semantically identical, but

the downstream classifier was indexing on urgency and silently fell

through to a default value of "low" on 100% of new tickets.

Nobody saw it because:

• The prompt didn't error. JSON parsed. Fields existed.


• The unit tests checked the prompt string, not the prompt output.


• The integration tests mocked the LLM call.


• The output was indistinguishable from "everything's fine and quiet."

This is the silent regression problem. Code has tests; prompts have vibes.

Three categories of model-swap failure

After looking at a dozen of these incidents, the failures cluster into three

groups. Knowing which kind you're looking at tells you what to test.

1. Format drift. The model decides to rename a field, drop a field, add

a field you didn't ask for, or change list ordering. JSON still parses. Your

downstream code breaks.

2. Reasoning regression. The model is "improved" but loses a hidden

constraint your prompt depended on. Classic example: GPT-4 reliably extracted

all requirements from a contract; GPT-4-Turbo extracted "the most important

ones," dropping 15-20% of clauses. The format was fine. The data was wrong.

3. Tone shift. Less common but expensive. The new model's outputs are

more verbose, less verbose, friendlier, blunter. If anything downstream

(another model, a regex, a fuzzy matcher) was tuned to the old tone, it

breaks.

What the team should have had

A test suite of 30 representative tickets, each with an expected JSON shape.

On model swap day:

$ promptfork test summarize_ticket --baseline gpt-4o

→ running v12 across [gpt-4.1] vs baseline [gpt-4o]

✗ 30/30 ok, but 6 regressions detected

  - urgency_field_renamed: 6 cases

  - severity 2 (functional)

Six lines. Seven seconds. Two-week customer-facing bug avoided.

How to actually do this

The setup for the team that got bitten took four minutes:

pip install promptfork



# Save the current production prompt, version 1

promptfork push summarize_ticket \

  --file prompts/summarize.txt \

  --message "current prod"



# Pin 30 real tickets from your support inbox

for t in tickets/*.json; do

  name=$(basename "$t" .json)

  promptfork add-test summarize_ticket "$name" \

    --input ticket="$(cat "$t")" \

    --rubric "must return urgency in {low,medium,high}"

done



# Run baseline on 4o

promptfork test summarize_ticket --models gpt-4o



# Now upgrade — push the new prompt as v2 (or keep v1 and swap models)

# Run with v1 (4o) as the baseline, get an LLM-judge regression report

promptfork test summarize_ticket --baseline 1 --models gpt-4.1

That's it. The --baseline flag is what catches drift — it pulls the

baseline output, runs the candidate, and asks Claude Haiku to compare them

under a strict "only flag strictly worse" rubric.

The CI version

The same command in a GitHub Action means no prompt change ever ships

without running against a known-good baseline:

- uses: shaunvand/promptfork-cli@v0

  with:

    prompt: summarize_ticket

    baseline: 1

    api-key: ${{ secrets.PROMPTFORK_API_KEY }}

The action exits non-zero on regression. Branch protection blocks the merge.

If you ship LLM features, you need this. The first time it catches a silent

regression, it pays for itself a hundred times over. PromptFork has a free

tier (3 prompts, 50 runs/mo) at https://promptfork.online/diff — set it up

in five minutes, sleep better forever.