Evaluating LLM Output Quality In Production

In March 2023, GPT-4 could tell you whether a number was prime with 97.6% accuracy. By June of the same year, the same model name answered those same questions correctly 2.4% of the time. Nobody pushed a bad commit. No prompt changed in your repo. The thing behind the API just... moved.

That number comes from a Stanford and Berkeley study, "How is ChatGPT's behavior changing over time?", and it's the cleanest illustration I know of the core problem with LLMs in production: the model is not a function you control. It's a dependency that can drift under your feet, silently, between two Tuesdays. And the only way you find out is if you're measuring, or if your users tell you first, which is the version where you lose.

So let's talk about how you actually measure the quality of LLM output once it's live, not in a demo. Not vibes. Not "looks good to me." A real, repeatable signal you can put a number on and gate a deploy with.

The thing that makes this hard

Traditional software is deterministic. Same input, same output, every time. You write assertEqual(add(2, 2), 4) and that test means something forever.

LLMs break that contract in two directions at once. First, the output is non-deterministic: ask the same question twice and you can get two differently-worded answers, both correct. So exact-match assertions are mostly useless; "4" and "The answer is 4." are the same answer to a human and different strings to your test runner. Second, "correct" is fuzzy. For a summarization feature there are a thousand acceptable summaries and no canonical one. You can't diff against a golden string when the golden string doesn't exist.

That's why you can't just port your unit-testing instincts over. You need evaluation that scores meaning and behavior, tolerates surface variation, and runs both before you ship and continuously after. Roughly, the job splits into three layers:

• 
  Offline evals: a fixed test set you run on every prompt or model change, like regression tests.


• 
  Reference-free checks: signals you can compute on a live answer with no "right answer" to compare against, like hallucination detection.


• 
  Production monitoring: watching the stream of real traffic for drift, refusals, and quality drops over time.

Get all three and you have a safety net. Skip any one and there's a hole the bad outputs fall through.

Golden datasets: your regression baseline

A golden dataset is the LLM version of a regression test suite. It's a curated, version-controlled set of inputs paired with expected outputs (or rubrics for grading them). Every time you touch the prompt, swap the model, or bump a temperature, you run the whole set and compare the score to the last known-good run. If groundedness drops three points, you found out in CI instead of in a support ticket.

The "golden" part matters. These aren't random production samples. They're hand-curated, reviewed examples that represent the cases you actually care about, including the weird ones. The edge cases are the whole point: the empty input, the adversarial prompt, the question in the second language your product half-supports, the one customer whose data breaks your parser. A golden set of 80 sharp examples beats 8,000 random ones, because the random ones cluster around the easy middle and tell you nothing about the failures that hurt.

Here's the shape of an offline eval. Same idea in a couple of languages, because eval harnesses live in both ecosystems:

:::tabs

eval_golden.py

import json



# golden.jsonl: one {"input": ..., "expected": ..., "rubric": ...} per line

def load_golden(path):

    with open(path) as f:

        return [json.loads(line) for line in f]



def run_eval(model, scorer, dataset, threshold=0.85):

    scores = []

    for case in dataset:

        output = model(case["input"])

        score = scorer(output, case)  # 0.0 - 1.0

        scores.append(score)

    avg = sum(scores) / len(scores)

    # fail the build if quality regressed below the bar

    assert avg >= threshold, f"quality {avg:.3f} below {threshold}"

    return avg

eval_golden.ts

import { readFileSync } from "node:fs";



type Case = { input: string; expected?: string; rubric?: string };



function loadGolden(path: string): Case[] {

  return readFileSync(path, "utf8")

    .trim()

    .split("\n")

    .map((line) => JSON.parse(line) as Case);

}



async function runEval(model: Model, scorer: Scorer, dataset: Case[], threshold = 0.85) {

  const scores: number[] = [];

  for (const c of dataset) {

    const output = await model(c.input);

    scores.push(await scorer(output, c)); // 0..1

  }

  const avg = scores.reduce((a, b) => a + b, 0) / scores.length;

  if (avg < threshold) throw new Error(`quality ${avg.toFixed(3)} below ${threshold}`);

  return avg;

}

:::

The interesting question isn't the loop, it's the scorer. What does it mean to score an output from 0 to 1? That's where most of the real work is.

Watching production without staring at it

Offline evals tell you a change is safe before you ship. They say nothing about the model drifting three weeks later, or a provider silently swapping the weights behind the endpoint, or your own prompt slowly rotting as five people each tweak it without writing down why. That last one even has a name now, prompt drift, the accumulated quality loss from small unrecorded edits, and it's blamed for a large share of production LLM incidents.

You don't need to grade every live request with an expensive judge. Layer it by cost:

• 
  Cheap heuristics on 100% of traffic. Parse-failure rate, output length distribution, refusal-phrase frequency ("I'm sorry," "I cannot"), latency, empty responses. These are nearly free and a sudden spike is the earliest possible warning. A jump in refusals or malformed JSON usually means the model or the API changed under you, exactly the silent-drift case from the intro.


• 
  Sampled LLM-judge on a slice. Run faithfulness and relevance scoring on, say, 1 to 5% of real traffic. Track the rolling average. When it dips below baseline, alert. You're not grading everything, just enough to see the trend move.


• 
  A drift baseline. Keep a reference distribution, score histograms from a known-good period, and compare today's sample against it. When the shape shifts, something changed even if no single answer looks obviously broken.

ci-quality-gate.sh

# block the deploy if the golden-set score regressed

python eval_golden.py --dataset golden.jsonl --threshold 0.85 \

  || { echo "::error::LLM quality regression - blocking deploy"; exit 1; }

The mindset shift that makes all this click: stop treating model quality as a fixed property you verified once at launch, and start treating it as a metric you monitor like latency or error rate. It moves. Your job isn't to prove it's good. It's to notice the moment it stops being good, ideally before anyone files a ticket.

Where to start if you have none of this

You don't build all three layers in a sprint. Start with the one that pays off fastest: sit down and write twenty golden examples. Just twenty: your worst real cases, the inputs you're scared of, the ones a customer already complained about. Wire them into CI with a single LLM-judge scoring faithfulness, and gate your deploys on the average. That alone catches the prompt edit that looked harmless and broke a quarter of your answers.

Then add the cheap production heuristics, because they're nearly free and they're what would've caught that 97.6%-to-2.4% cliff on day one instead of in the news. The fancy stuff, semantic entropy, drift baselines, calibrated judge panels, comes later, when you've felt enough pain to know exactly which hole you're plugging.

The teams that sleep well aren't the ones with the smartest model. They're the ones who'd know within an hour if it got dumber overnight. Be that team.

Originally published at nazarboyko.com.