Hallucination Detection Is Not a Model Problem—It's an Architecture Problem

Hallucination Detection Is Not a Model Problem—It's an Architecture Problem

Every week someone publishes a new paper on reducing hallucination rates by 3% with a better prompt or a fancier retrieval strategy. Meanwhile, in production, your agent just confidently told a customer their refund was processed when it wasn't.

The problem isn't that models hallucinate. The problem is that your system has no architectural mechanism to catch hallucinations before they reach users.

Stop treating hallucination as a model tuning problem. Start treating it as a systems observability problem.

The Three Layers of Production Hallucination

In my experience running agents in production, hallucinations fall into three categories:

1. 
   Factual fabrication — the model invents data that doesn't exist in context


2. 
   Action hallucination — the model claims it performed an action it didn't


3. 
   State hallucination — the model misrepresents the current state of the system

Most teams only think about #1. But #2 and #3 are where production incidents actually live. Your agent says "I've updated the database" when the tool call failed silently. Your agent says "The deployment is healthy" when it never checked.

These aren't prompt engineering problems. They're verification architecture problems.

The Verification Layer Pattern

The fix is straightforward: insert a verification layer between your agent's claims and your system's outputs. Every claim the agent makes about the world should be independently verifiable.

Here's what this looks like in practice:

import { AgentOutput, VerificationResult } from "./types";



interface ClaimExtractor {

  extract(output: AgentOutput): Claim[];

}



interface Claim {

  type: "factual" | "action" | "state";

  content: string;

  verifiable: boolean;

  verificationStrategy?: "tool-replay" | "state-check" | "grounding-match";

}



interface VerificationLayer {

  verify(claims: Claim[], context: ExecutionContext): Promise<VerificationResult[]>;

}



export class HallucinationDetector {

  constructor(

    private extractor: ClaimExtractor,

    private verifier: VerificationLayer,

    private threshold: number = 0.8

  ) {}



  async check(output: AgentOutput, context: ExecutionContext): Promise<DetectionResult> {

    const claims = this.extractor.extract(output);

    const verifiableClaims = claims.filter(c => c.verifiable);



    if (verifiableClaims.length === 0) {

      return { safe: false, reason: "no-verifiable-claims", claims };

    }



    const results = await this.verifier.verify(verifiableClaims, context);

    const verified = results.filter(r => r.status === "confirmed");

    const score = verified.length / verifiableClaims.length;



    return {

      safe: score >= this.threshold,

      score,

      claims,

      results,

      flagged: results.filter(r => r.status === "contradicted")

    };

  }

}

The key insight: if your agent cannot produce verifiable claims, that itself is a signal. An agent that says "I think the deployment might be okay" is actually more trustworthy than one that says "The deployment is healthy" without calling the health check endpoint.

Verification Strategies That Actually Work

Tool Replay for Action Claims

When your agent claims it performed an action, replay the tool call and verify the state change:

async function verifyActionClaim(

  claim: Claim,

  ctx: ExecutionContext

): Promise<VerificationResult> {

  const toolCalls = ctx.getToolCalls();

  const relevantCall = toolCalls.find(tc => tc.relatesTo(claim));



  if (!relevantCall) {

    return { status: "contradicted", reason: "no-matching-tool-call" };

  }



  if (relevantCall.result.error) {

    return {

      status: "contradicted",

      reason: "tool-call-failed",

      evidence: relevantCall.result

    };

  }



  // Verify the side effect actually occurred

  const stateAfter = await ctx.checkState(relevantCall.expectedEffect);

  return stateAfter.matches

    ? { status: "confirmed", evidence: stateAfter }

    : { status: "contradicted", reason: "effect-not-observed", evidence: stateAfter };

}

Grounding Match for Factual Claims

For factual claims, check whether the information exists in the context the agent was given:

async function verifyFactualClaim(

  claim: Claim,

  ctx: ExecutionContext

): Promise<VerificationResult> {

  const sources = ctx.getRetrievedDocuments();

  const groundingScore = await computeGrounding(claim.content, sources);



  if (groundingScore > 0.9) return { status: "confirmed", groundingScore };

  if (groundingScore < 0.3) return { status: "contradicted", groundingScore };

  return { status: "uncertain", groundingScore };

}

The Observability Angle

Detection without observability is useless. You need to track:

• 
  Hallucination rate by claim type — are action hallucinations increasing?


• 
  Verification coverage — what percentage of agent outputs have verifiable claims?


• 
  Time-to-detection — how long between hallucination and catch?


• 
  Escape rate — hallucinations that reached users before detection

This is where most teams fail. They build a detector but don't instrument it. Three weeks later, they have no idea if their hallucination rate is getting better or worse.

The Uncomfortable Truth

Here's what nobody wants to hear: you cannot solve hallucination at the model layer alone. Even a model with a 0.1% hallucination rate will hallucinate thousands of times per day at production scale.

The question isn't "how do I make my model hallucinate less?" The question is "how do I architect my system so hallucinations get caught before they matter?"

This means:

• Every agent output goes through claim extraction


• Every verifiable claim gets verified


• Every unverifiable output gets flagged for human review or conservative fallback


• Every detection event feeds back into your eval suite

From Detection to Prevention

The final piece: your hallucination detection data should feed directly into your evaluation infrastructure. Every caught hallucination becomes a test case. Every pattern of hallucination becomes a regression suite.

This is where tools like agent-eval become critical — they let you turn production hallucination events into deterministic eval cases that run on every deployment. And if you're looking for the observability layer to track these patterns over time, AgentLens gives you the dashboards and alerting you need to know when your hallucination rate is drifting before users notice.

Hallucination isn't a bug to fix. It's a failure mode to architect around.

Build the verification layer. Instrument it. Let your evals learn from production. That's how you ship agents that don't lie.