I Benchmarked 19 Retrieval Pipelines Head-to-Head and the Results Were Surprisingly Honest

I got tired of retrieval papers claiming their pipeline is "state-of-the-art" without significance tests, without telling you what dataset they ran on, and without reproducible code. So I built raven-retrieval -- 19 retrieval pipelines, real BEIR datasets, Bonferroni-corrected bootstrap significance tests, all the numbers whether they look good or not. MIT licensed, open source.

What It Does

The core question: does ColBERT-style late interaction scoring at every level of a RAPTOR hierarchical tree actually improve retrieval over simpler approaches? To answer that properly you need the same datasets, same chunking, same metrics, same significance tests for every pipeline. Otherwise you're comparing apples to oranges.

The framework implements 19 pipelines and benchmarks them all on BEIR datasets (SciFact, HotpotQA) on equal footing. No cherry-picking.

The 19 Pipelines (Quick Summary)

Core: Naive Dense RAG, Hybrid RAG (BM25+Dense+RRF), ColBERT Late Interaction, RAPTOR+Late Interaction (novel combination).

Research (2024-2025): HyDE, SPLADE, SPLADE+Dense, BM25+Rocchio PRF, Contextual Retrieval, Contextual Hybrid, Late Chunking, RAPTOR+Late Traversal, Agentic Multi-Hop, Reflection Retriever, Graph Retrieval, Two-Stage Dense+Reranker, Approximate Late Interaction, Contextual Dense, Contextual BM25.

Composition modules: Cross-Encoder Reranker, Two-Stage Retriever, Document Graph, Residual Compressor (~30x storage reduction).

The novel one is RAPTOR + Late Interaction: build a RAPTOR hierarchical summary tree, then apply ColBERT MaxSim scoring at every node (leaf chunks AND summary nodes). Nobody has published on this combination before.

How MaxSim Works

ColBERT's scoring: for each query token, find the most similar document token (cosine), then sum those maximums across all query tokens. Every query token gets a vote for its best-matching document token. This captures fine-grained token-level relevance that single-vector cosine misses entirely. Implemented in pure numpy, no torch needed for scoring.

The Benchmark Results

Run on Google Colab T4 GPU, seed=42, two datasets.

SciFact (100 queries, single-hop scientific claims)

Key finding: Top 4 are statistically indistinguishable (all pairwise p > 0.05 after Bonferroni). BM25+PRF is significantly worse (p ~ 0.000). The "winner" HyDE is only 2 points above Dense -- within noise.

HotpotQA (50 queries, multi-hop reasoning)

The ranking flips completely. HyDE drops from 1st to 4th. Hybrid RAG wins because BM25+Dense fusion catches different reasoning hops. A single hypothetical document can't bridge two separate reasoning steps.

The most important finding: No single pipeline dominates across task types. Single-hop favors semantic (HyDE). Multi-hop favors hybrid. Anyone claiming universal superiority hasn tested on enough datasets.

The Bugs I Found

Before v0.3, 13 bugs (6 critical) silently corrupted results:

• 
  ColBERT matched against PAD tokens -- attention mask wasn used, padding tokens got MaxSim votes alongside real content


• 
  RAPTOR clustering used wrong embedding space -- centroids from ColBERT means while scoring with SBERT vectors


• 
  Per-query nDCG was all zeros -- BEIR evaluate() returns averaged floats, not per-query; significance tests fed zeros and reported "no difference" everywhere (false negatives)


• 
  Reflection evaluated keyword coverage over doc ID strings ("doc_1", "doc_42") instead of actual text, triggering endless reformulation loops


• 
  RAPTOR pipelines crashed -- UMAP bug when local clusters had 3 samples (k >= N), burned 189 minutes of Colab time

These don crash your program. They just make your results wrong in ways you can see from final numbers alone.

Known Limitations

• ColBERT projection head is Xavier-initialized (untrained). Late interaction scores ~0.58 without training, worse than Dense at 0.696. Training support exists (ColbertContrastiveEncoder with InfoNCE) but no trained checkpoint in default benchmark yet.


• GMM soft-clustering converges to near-hard assignment on ~100-token chunks (matches Stanford CS224N reproduction).


• Late Chunking limited by BERT's 512-token window -- needs long-context model to really shine.


• HotpotQA scores are on 2,000-doc subsample, not the full 5.2M corpus. Real retrieval is harder than these numbers suggest.
  

git clone https://github.com/subhansh-dev/raven-retrieval.git

cd raven-retrieval

pip install -e ".[full]"

python run_enhanced_benchmark.py --dataset scifact --top-k 10 --seed 42

CLI entry points: raven-benchmark, raven-report. Low-RAM? Use --max-docs. Want trained ColBERT? Use --colbert-checkpoint. All 55 tests pass.

Why This Matters

Retrieval benchmarks have a credibility problem. Papers claim improvements without significance tests, without multiple datasets, without reproducible code. Raven-retrieval fixes all of that. The honest answer -- that top pipelines are tied, that rankings flip between datasets, that BM25+PRF is bad on scientific text -- is more useful than cherry-picked claims of universal superiority.

github.com/subhansh-dev/raven-retrieval | subhansh.dev