Hallucination Detection Is Not a Model Problem—It's an Architecture Problem
Every week someone publishes a new paper on reducing hallucination rates by 3% with a better prompt or a fancier retrieval strategy. Meanwhile, in production, your agent just confidently told a customer their refund was processed when it wasn't.
The problem isn't that models hallucinate. The problem is that your system has no architectural mechanism to catch hallucinations before they reach users.
Stop treating hallucination as a model tuning problem. Start treating it as a systems observability problem.
The Three Layers of Production Hallucination
In my experience running agents in production, hallucinations fall into three categories:
Factual fabrication — the model invents data that doesn't exist in context
Action hallucination — the model claims it performed an action it didn't
State hallucination — the model misrepresents the current state of the system
Most teams only think about #1. But #2 and #3 are where production incidents actually live. Your agent says "I've updated the database" when the tool call failed silently. Your agent says "The deployment is healthy" when it never checked.
These aren't prompt engineering problems. They're verification architecture problems.
The Verification Layer Pattern
The fix is straightforward: insert a verification layer between your agent's claims and your system's outputs. Every claim the agent makes about the world should be independently verifiable.
Here's what this looks like in practice:
import { AgentOutput, VerificationResult } from "./types";
interface ClaimExtractor {
extract(output: AgentOutput): Claim[];
}
interface Claim {
type: "factual" | "action" | "state";
content: string;
verifiable: boolean;
verificationStrategy?: "tool-replay" | "state-check" | "grounding-match";
}
interface VerificationLayer {
verify(claims: Claim[], context: ExecutionContext): Promise<VerificationResult[]>;
}
export class HallucinationDetector {
constructor(
private extractor: ClaimExtractor,
private verifier: VerificationLayer,
private threshold: number = 0.8
) {}
async check(output: AgentOutput, context: ExecutionContext): Promise<DetectionResult> {
const claims = this.extractor.extract(output);
const verifiableClaims = claims.filter(c => c.verifiable);
if (verifiableClaims.length === 0) {
return { safe: false, reason: "no-verifiable-claims", claims };
}
const results = await this.verifier.verify(verifiableClaims, context);
const verified = results.filter(r => r.status === "confirmed");
const score = verified.length / verifiableClaims.length;
return {
safe: score >= this.threshold,
score,
claims,
results,
flagged: results.filter(r => r.status === "contradicted")
};
}
}
The key insight: if your agent cannot produce verifiable claims, that itself is a signal. An agent that says "I think the deployment might be okay" is actually more trustworthy than one that says "The deployment is healthy" without calling the health check endpoint.
Verification Strategies That Actually Work
Tool Replay for Action Claims
When your agent claims it performed an action, replay the tool call and verify the state change:
async function verifyActionClaim(
claim: Claim,
ctx: ExecutionContext
): Promise<VerificationResult> {
const toolCalls = ctx.getToolCalls();
const relevantCall = toolCalls.find(tc => tc.relatesTo(claim));
if (!relevantCall) {
return { status: "contradicted", reason: "no-matching-tool-call" };
}
if (relevantCall.result.error) {
return {
status: "contradicted",
reason: "tool-call-failed",
evidence: relevantCall.result
};
}
// Verify the side effect actually occurred
const stateAfter = await ctx.checkState(relevantCall.expectedEffect);
return stateAfter.matches
? { status: "confirmed", evidence: stateAfter }
: { status: "contradicted", reason: "effect-not-observed", evidence: stateAfter };
}
Grounding Match for Factual Claims
For factual claims, check whether the information exists in the context the agent was given:
async function verifyFactualClaim(
claim: Claim,
ctx: ExecutionContext
): Promise<VerificationResult> {
const sources = ctx.getRetrievedDocuments();
const groundingScore = await computeGrounding(claim.content, sources);
if (groundingScore > 0.9) return { status: "confirmed", groundingScore };
if (groundingScore < 0.3) return { status: "contradicted", groundingScore };
return { status: "uncertain", groundingScore };
}
The Observability Angle
Detection without observability is useless. You need to track:
Hallucination rate by claim type — are action hallucinations increasing?
Verification coverage — what percentage of agent outputs have verifiable claims?
Time-to-detection — how long between hallucination and catch?
Escape rate — hallucinations that reached users before detection
This is where most teams fail. They build a detector but don't instrument it. Three weeks later, they have no idea if their hallucination rate is getting better or worse.
The Uncomfortable Truth
Here's what nobody wants to hear: you cannot solve hallucination at the model layer alone. Even a model with a 0.1% hallucination rate will hallucinate thousands of times per day at production scale.
The question isn't "how do I make my model hallucinate less?" The question is "how do I architect my system so hallucinations get caught before they matter?"
This means:
- Every agent output goes through claim extraction
- Every verifiable claim gets verified
- Every unverifiable output gets flagged for human review or conservative fallback
- Every detection event feeds back into your eval suite
From Detection to Prevention
The final piece: your hallucination detection data should feed directly into your evaluation infrastructure. Every caught hallucination becomes a test case. Every pattern of hallucination becomes a regression suite.
This is where tools like gives you the dashboards and alerting you need to know when your hallucination rate is drifting before users notice.
Hallucination isn't a bug to fix. It's a failure mode to architect around.
Build the verification layer. Instrument it. Let your evals learn from production. That's how you ship agents that don't lie.
SOCIAL SHARE CARD GENERATOR