Retrieval-Augmented Self-Recall — Part 4: Benchmarking Retrieval *and* Honesty

Part 4 of Retrieval-Augmented Self-Recall. Code: .

A standard RAG benchmark would have handed my retriever a perfect score on a day it was confidently answering questions it had no data for.

That's not a bug in my system. It's a gap in what those benchmarks ask. Every one of them asks the same thing: when there was an answer, did you rank it first? None of them ask the question that decides whether agent memory is safe: when there was no answer, did you say so?

So RE-call ships its own harness. Here's how it works, and the first finding it produced.

The test set: the unanswerable queries are the point

The evaluation runs on 14 answerable queries + 5 unanswerable queries over a synthetic corpus.

Those 5 unanswerable queries are the whole reason the harness exists. They're questions the corpus genuinely cannot answer, where the correct behavior is to abstain — to fire gap_warning, not to confidently return the nearest memo. Standard retrieval benchmarks are built entirely from answerable queries; they have no way to score "did it correctly say nothing?" This harness is built around that case.

Two families of metrics

Because there are two jobs — rank well when there's an answer, abstain when there isn't — there are two families of metrics:

Ranking quality (for the answerable queries):

• precision@k, recall@k


• MRR (mean reciprocal rank)


• nDCG@10

Guard quality (for the unanswerable queries):

• 
  False-confident rate (FCR) — the fraction of unanswerable queries that the guard failed to flag. High FCR means the system confidently answered questions it should have abstained on. This is the honesty metric, and it's the one almost nobody reports.

Why you need both is the crux: a system can post excellent MRR and terrible FCR. It ranks beautifully whenever an answer exists, and lies confidently whenever one doesn't. If you only look at ranking metrics — as most RAG evals do — that failure is completely invisible. FCR is what drags it into the light.

The ablation: every embedder × every fusion stage

The harness runs the full matrix: each embedder (HashingEmbedder, bge-small, voyage-3) crossed with each fusion configuration (dense only → hybrid → hybrid + rerank). That's what lets you answer "which component actually earns its cost?" instead of cargo-culting a reranker into every pipeline.

And it runs against the real thing: 49 integration tests on a live pgvector container (of a 150-test suite), in CI, no mock database. The benchmark exercises the actual retrieval path, not a stand-in.

Finding 1: hybrid + rerank helps most exactly where you'd expect — and nowhere else

Here's the ablation on the weak (hashing) embedder — quality climbs monotonically as you add stages:

Now the same pipeline on the strong bge-small embedder: dense retrieval already achieves a perfect .)

Part 4 of Retrieval-Augmented Self-Recall. Code: .