RAG for developers who aren't AI engineers: what actually matters

Most non-AI developers have a mental model of RAG that is either wrong or dangerously incomplete. Not because they're bad engineers — because almost everything written about RAG is either a 10-minute framework tutorial that hides every real decision, or a research paper aimed at people who already do this for a living. There's very little in between.

This article is the in-between. I recently built a production RAG system over 62 ancient-history books (~46,000 chunks) and measured every design decision against a fixed test set — including the decisions that didn't work. I'm going to skip the framework marketing and tell you the small number of things that actually determine whether your RAG system is trustworthy, with the numbers to back them up.

You don't need ML knowledge to follow this. If you can build a REST API and query Postgres, you can build everything described here.

The 60-second mental model

RAG (Retrieval-Augmented Generation) is just this:

question → find relevant text in your corpus → paste it into the LLM prompt → "answer ONLY from this text"

That's the whole idea. A search engine bolted onto an LLM, with instructions to answer from the search results instead of from its training data.

The only genuinely new component for a typical backend developer is embeddings. An embedding model turns text into a vector (an array of ~1,000 floats) where similar meaning produces nearby vectors. So "find relevant text" becomes nearest-neighbor search over vectors: embed all your document chunks once at ingest time, embed the user's question at query time, fetch the k closest chunks.

If you know Elasticsearch: it's like that, but matching on meaning instead of keywords. "Why did the ship sink" will match a paragraph about "the vessel foundered in the storm" even though they share zero words.

Everything else — chunking, retrieval, prompting — is ordinary engineering. Which is exactly why regular developers can and should build these systems. What's missing from most tutorials isn't skill, it's knowing where the traps are.

The trap: naive RAG works in the demo and lies in production

Here's the part that shocks people.

The standard tutorial setup — split documents into fixed 500-token chunks, embed with whatever the tutorial used, retrieve top-5, feed to the model — appears to work immediately. You ask a question, you get a fluent, confident, well-structured answer with a citation. Demo done, ship it, right?

I measured that exact baseline on my corpus with a 135-question test set where I knew which passages contained each answer. Result: recall@5 = 35.2%.

Read that again. For roughly two out of three questions, the correct passage never reached the model at all. And here's the dangerous part: the model answered anyway. Fluently. With the same confident tone as when the retrieval worked.

This is the core thing non-AI developers get wrong about RAG. The failure mode is not "no results found" like a search engine, and not an exception like an API call. The failure mode is a confident, plausible fabrication that is indistinguishable from a correct answer unless you already know the answer. A system that's right 65% of the time and silently makes things up the other 35% is worse than useless — because you can't tell which answer you're looking at.

Nothing about the demo tells you this is happening. The only way to know is to measure. Which brings us to the single most important piece of advice in this article.

Before you improve anything: build a test set

This is the step every beginner tutorial skips, and it's worth more than any technique you'll ever add.

Before touching chunk sizes, embedders, rerankers, or agent frameworks, do this:

1. Write 30–50 real questions about your corpus — the kind your actual users will ask, not the kind your documents happen to answer neatly.


2. For each question, record where the answer lives (which document, roughly which passage).


3. Crucially, include 5–10 questions your corpus cannot answer. These are your hallucination traps. The correct behavior is a refusal; anything else is fabrication you'd otherwise never see.


4. Measure two things separately:
   
   
   
   
   • 
     Retrieval: is the right passage in the top-k results? (Pure code, no LLM judging needed — this is the number to optimize first.)
   
   
   • 
     Generation: given the right passages, is the answer grounded in them? Does the system refuse when it should?
   
   
   
   

This costs one or two days and it changes everything, because RAG advice on the internet is wildly corpus-dependent. Things that transformed one system do nothing on another. Without your own test set, every blog post — including this one — is a coin flip. With it, you can check any technique against your data in an hour.

Concrete example of why this matters: the standard 2024-era advice says "hybrid search (keyword BM25 + vector) always beats pure vector search at scale." I believed it. I expected it to win on my corpus and pre-registered that prediction. When I actually ran it, hybrid retrieval was byte-identical to plain dense retrieval — same recall, category by category, not one result changed. I only know that because I measured. (The full write-up of that negative result is its own story.)

Your test set is not test infrastructure you write once and forget. It's the steering wheel.

What actually moved the needle (ranked)

With a test set in place, here's what mattered on my corpus, in order of impact.

1. The embedding model — the biggest single lever

Swapping the default embedder for a strong one (qwen3-embedding-8b, via a hosted API) took recall@5 from 35% to 53% — an 18-point jump from changing one line of configuration. The hardest-hit category improved by +41.7 points: questions phrased in modern English against text written in formal Victorian translation. The weak embedder simply couldn't bridge the vocabulary gap; the strong one could.

The lesson: don't inherit the tutorial's embedder. It's the component doing the actual "understanding" in your retrieval, models differ enormously, and swapping it is trivial. Check the (try asking it something the corpus can't answer — that's the interesting part)

It's Python, but nothing about the architecture is Python-specific — the structure maps one-to-one onto the TypeScript stack above.

I build RAG and LLM-evaluation systems, and I'm available for contract work. If your team is trying to make an LLM answer reliably from your own data — or trying to figure out whether the one you built already can be trusted — my case study above shows how I approach it, with published numbers. Reach out: [email protected].