0 % vs 50 %: making a RAG agent refuse to hallucinate
2026-05-31 · LLM / RAG
A retrieval-augmented agent is only as trustworthy as its behaviour on questions whose answer
isn't in the corpus. The failure mode is quiet: instead of saying "I don't know," the model
invents a confident, well-formed, wrong answer. This post shows a single guardrail that takes
that from common to never — and, crucially, measures it.
Reference architecture:
nim-agent-blueprint — agentic RAG on
the NVIDIA NIM stack with a built-in eval harness.
The ablation
The agent loop is plan → retrieve → generate → validate. The interesting variable is the
generation prompt's contract with the retrieved context:
| Configuration | Out-of-corpus hallucination rate |
|---|---|
| Generate freely from context | ~50 % |
| Guarded prompt (answer only from context; otherwise abstain) | 0 % |
Same model, same retriever, same questions. The only change is a prompt that makes "I can't
answer that from the provided sources" a first-class, rewarded output — plus a validate
step that checks the answer is grounded in retrieved spans before returning it. On in-corpus
questions, retrieval recall@3 stayed at 94–100 %, so the guardrail buys safety without
costing coverage.
Why "just prompt better" isn't the lesson
The lesson isn't the prompt — it's that the difference between 50 % and 0 % is invisible
without an eval harness. A demo that only asks in-corpus questions looks perfect in both
configurations. You only see the 50 % when you deliberately ask things the corpus can't
answer and score groundedness. So the blueprint ships with:
retrieval hit-rate (is the answer even retrievable?),
answer groundedness via LLM-as-judge (is the answer supported by what was retrieved?),
latency, and OpenTelemetry traces per agent step.
That's the difference between "it works on my five questions" and "here is the number a
partner can hold me to."
Takeaway
For enterprise RAG, abstention is a feature, not a failure. Make "I don't know" a rewarded
output, validate groundedness before returning, and measure the out-of-corpus rate — it's
the number that separates a demo from something you'd put in front of a customer.
→ Runnable blueprint + eval harness:
github.com/waynehacking8/nim-agent-blueprint
SOCIAL SHARE CARD GENERATOR