I have 312 docs in my personal knowledge DB. Tweets, arxiv abstracts, Zenn articles, blog posts, YouTube transcripts. Claude Code writes to it, reads from it, and cites out of it every day. That number is not a brag. It is the reason I finally have data on which retrieval strategy holds up in an LLM-native workflow.
I tried four. The one I ship is the one I tried last and expected to lose. Three of the four broke in ways that are worth naming, because the broken versions are what most tutorials will tell you to build.
The setup, so we agree on what got benchmarked
The knowledge DB is called context-forge internally. It is a folder, some markdown files, and a SQLite table. Claude Code adds to it via CLI, searches via CLI, and reads the underlying markdown directly when it needs the full text. It took eight hours to build the CLI, three months to accumulate the 312 documents at a pace of one to five per day, and about 15 minutes a day of my time to keep it flowing.
Each doc has metadata: source URL, a credibility score 1-5, one to three categories, a short summary. The autoregistration pipeline is Claude Code itself: I paste a URL, it fetches, summarizes, scores, categorizes, writes the markdown, commits, and updates the SQLite index. The pipeline is not the interesting part. The retrieval strategy is.
I ran each of the four strategies for two weeks against the same day-to-day tasks: writing a chapter, answering "what did that person say about X," and building an argument for a decision. Same me, same DB, different retriever.
Layer 1: pure semantic RAG (vector embeddings). Broke at 200 docs
The first version was the textbook answer. Embed every document with a sentence transformer, store the vectors in SQLite with a similarity index, retrieve the top-k on every query. This is the pattern .
It worked at 50 docs. It worked at 100. Around 200 documents it started retrieving noise. The top-5 by cosine similarity would return three barely-related X posts and skip a Zenn article I had explicitly written on the exact topic. The reason is not exotic: the DB is topically dense (LLM tooling, agent workflows, Claude Code) and short (2-3 paragraphs per doc), so vector similarity flattens. Everything looks 0.78 similar to everything else.
The other failure mode is worse. Claude Code would confidently cite the wrong document, because the retrieval returned it and the LLM assumed the retrieval was right. I did not catch this until I noticed a quote in a draft that did not exist in the source. This is the well-known retrieval hallucination pattern. I was demonstrating it live to myself for two weeks.
Broke because: short + topically dense docs have degenerate embedding neighborhoods, and the LLM cannot tell when retrieval failed.
Layer 2: keyword search over titles + summaries. Broke at 300 docs
The next attempt was the least clever thing that could work. SQLite FTS5 over the title, summary, and category fields, no embeddings. Ranked by BM25.
At 100 docs this was better than vector search. Precision jumped because the failure mode of "everything looks 0.78 similar" went away. If I searched "graph db," I got the three docs with those words in the title, and they were the right three.
It failed at 300 because English is polysemous and my queries got sloppy. "Agent" retrieved everything: LLM agents, browser agents, code-review agents, sales agents. "Skill" retrieved Claude Code Skills, D&D skill trees (my own note from an unrelated context-forge dump), and a marketing framework called "skill stack." I ended up rewriting queries three times to add disambiguating words that I already knew the doc had, which defeated the point.
Broke because: BM25 is precision-first, so you get the right doc only if your query has the exact word the author used, and I frequently do not remember that word.
Layer 3: Claude Code Skills over the folder. Broke because it worked too well
In late 2025 I moved the whole DB into a Claude Code Skill. The docs stay as markdown in a folder; the Skill describes the DB's structure and gives Claude Code a search-knowledge tool that runs both keyword and vector search and returns the top-k with credibility scores. is the general-purpose version of the same idea; my Skill is a domain-specific one.
This should have been the winner. And it kind of was: retrieval quality jumped, because Claude now had two retrieval methods, could pick between them, and could re-query if the first attempt looked wrong. Cited quotes stopped hallucinating. Category filtering became sharper because Claude could read the schema and craft queries against it.
The failure was different. It became too eager. Every question triggered a knowledge search, including questions where I already knew the answer and just wanted the model to help me phrase it. I would ask "how should I open this section" and get 40 seconds of tool calls searching for context I did not need. My time-per-response went up 3x on the easy cases and 1.2x on the hard cases, which was the wrong side of the trade-off.
Broke because: giving Claude the tool means Claude uses the tool, and the cost of the tool call is now on the critical path for every response, even the ones where retrieval was pointless.
Layer 4: hand-written index cards, keyword grep, Claude reads only what I hand it
This is the one I ship.
Every doc gets a one-line, hand-written summary card in a single file called INDEX.md. That file is 312 lines long now, 20-40 words per line. It lives at the root of the folder. Claude Code has this file in context by default. When I ask a question, Claude scans INDEX.md for relevant cards (this happens in the prompt, no tool call), picks the two or three most relevant, and only then reads the underlying full-text markdown for those specific docs.
Two things about this. First, the index is written by me, so the language matches how I actually query. "Graph DB" in the index is the same "graph DB" I would type, because I wrote both. Second, the retrieval is single-hop and in-context: Claude reads INDEX.md, decides, reads the specific docs. No embedding search, no MCP round-trips, no tool-choice cost.
The cost is that I have to write the index card when I ingest the doc. That is 30 seconds of my time per doc, and I do it during the daily 15-minute ingestion window. In exchange, the retrieval is fast, the citations are always real (because the model reads the full doc before citing), and the tool never returns three unrelated X posts.
The DB is 312 docs. INDEX.md at 8000 tokens fits in a normal Claude Code context window with plenty of room. If the DB grew to 3000 docs, I would need to shard by category. It is not clear I ever need 3000, because I have a working DB at 312.
Held because: the retrieval reads a human-curated summary in a single pass and only fetches full text for the survivors. It is the same pattern Google Search's snippets use, and it works for the same reason: a good short summary is a good filter.
The comparison, in the table I wish I had at the start
| Layer | What it is | Best point | Where it broke | Days I lasted |
|---|---|---|---|---|
| 1. Vector RAG | Embed all docs, cosine top-k | High recall in the sparse case | Neighborhood collapse at ~200 docs; silent wrong citations | 14 |
| 2. BM25 keyword | SQLite FTS5, ranked | Precision on named entities | Query terms I could not remember; polysemy blowup | 14 |
| 3. Skill + hybrid | Claude picks retrieval method | Cited quotes real; category filtering sharp | Tool-choice tax on every prompt; over-eager retrieval | 21 |
| 4. Hand-written INDEX.md | Curated one-liners in context | Fast, cheap, always cites real text | Requires 30s of human work per doc | shipping |
Three broke. One held. The one that held is small.
What the failures agree on
The three that broke all fail on the same underlying assumption: that retrieval is a cheap oracle and the LLM can trust its output. It is not, and it cannot. Vector search fails silently. BM25 fails when your vocabulary drifts. Automated hybrid search fails by being called too often. Every one of these was fixable by narrowing what the retriever is allowed to do, which is why the hand-written index wins: it is the version where the retrieval budget is spent on the parts that need it and nothing else.
If you already have Claude Code Skills or MCP Memory set up and it is working for you at your scale, great, keep it. If you are 200 docs in and starting to notice hallucinated citations or expensive tool loops, the index-card version fits in an afternoon and might buy you three more months of not rebuilding the whole thing.
What I would not recommend
Do not skip building the DB because you think Claude's memory feature will cover it. . The chapter is called "knowledge automation" and it covers what I would build next if I hit 3000 docs.
SOCIAL SHARE CARD GENERATOR