The benchmark that built the tools

How we measured a markdown knowledge graph as agent memory — lost to grep, rebuilt our search and editing primitives, retracted our own best number, and ended with a $4.50 curator whose store reads back at 96% of a hand-built ceiling in a single call.

The question

Every AI agent product eventually hits the same wall: the model forgets. Context windows end, sessions restart, and everything the agent learned about you evaporates. An industry has grown up around this — hosted "memory layers" that ingest your conversations, extract facts with an LLM, embed them into a vector store, and sell retrieval back to you by the API call.

)" where the curator had been told, in three successive prompt versions, to write "(" — so the hub line itself answers when, and the link still points at the evidence. The other three were instance confusion: which tournament win was which, fixable by numbering the instances in the line ("won her second tournament…") so counting questions read straight off the ledger.

The date-as-link-text pattern has a satisfying property: it was impossible to produce until the renderer fix landed days earlier — the old write path would have overwritten the event date with a page title on the next save. The product change didn't just remove bloat; it made a new curation grammar expressible, and the prompt's job shrank to teaching it.

It closed half of them. The next run's curator adopted all three patterns from its first session, temporal recovered from 0.75 to 0.85, and the overall score reached 0.778 — the best automated number of the arc, 96% of the hand-built ceiling, from a curator that costs $4.45 a conversation. The progression tells the story of the whole guards era in one line: 0.68 with prompt rules alone, 0.70 after three rounds of prompt whack-a-mole, 0.77 the day the lint landed, 0.778 once the renderer defect was fixed in the product and the prompt could finally teach the pattern. What separates the automated store from the hand-built one is down to about seven questions of 199. Repeated answering runs over a fixed store differ by about two questions, so the gap is real — it is just, finally, small.

What we actually learned

About memory. Capture is the easy part — every configuration we ever ran had the facts somewhere. What separates configurations is dating discipline (store uncertainty, not false precision), co-location (facts that answer questions together must live together), and dated status lines for evolving state. Temporal questions, the weakest category for every published memory system, went from 0.60 to 1.00 across the dev rounds on prompt rules alone.

About tools for agents. Output rendering is a cost model, not a formatting choice. Guards beat instructions — an expect clause or a key-collision error catches what a prompt admonition never will, and the same holds at store scale: key formats, page budgets, and near-duplicates are checkable policy, not prompt hopes. Error messages are load-bearing documentation. And enforcement should live at the surface, not the grammar: humans get freedom, agents get strictness, the language means the same thing everywhere.

About enforcement layers. The guards sorted themselves into a taxonomy: schemas own shape (sections, budgets, block types — hard gates, nothing to argue with), graph hygiene owns links (dangling, orphans, near-duplicates — warnings that re-fire until fixed), and prompts own semantics (which date is the event's, what deserves a timeline line — the parts no pattern can check). Two laws attach. Every threshold must be calibrated against a known-good store before it ships — each one we tried flagged the good store first — while structural rules were robust on arrival. And agents negotiate with warnings but cannot negotiate with a refused write, so put the rules you cannot afford to lose behind gates and let the rest apply pressure.

About cost. One-time curation amortizes to nothing; per-question costs compound forever. The configuration of record — about $4.50 to curate a months-long relationship into a store, then one-shot answering at ~$0.05 a question — reflects that arithmetic. And raw-transcript grep, the baseline that never stopped embarrassing us, carries a hidden divergence: real conversation history grows without bound, while a curated store stays compact. (The strongest no-curation configuration in our records — IWE's own tools pointed at the raw transcript files — scores 0.80–0.85 across dev runs, and rides the same growth curve.) The benchmark's tiny corpora understate the curated arm's advantage, not overstate it.

About method. Until the comparison section below, every number here is a development number — twenty-question slices in the early rounds, one 199-question conversation in the late ones, single repetitions unless stated, an LLM judge with known artifacts (two of our persistent "failures" are answer-key defects we deliberately did not tune around). The test set's history is messier than the design, and we would rather record it than smooth it: the first installment was spent on the multi-turn configuration and told us grep wins; one of those spent conversations was then recycled as the late-arc development conversation; the second installment ran two conversations sealed until first contact, and their sealed-only score matches the overall number. Four test conversations remain unspent, reserved for the frozen configuration. We publish the harness, the prompts, the judge verbatim, the full run ledger — including the quarantined runs behind the retraction — and the per-question failure classifications precisely so the numbers can be distrusted productively.

The benchmark set out to test a thesis and kept disproving successive versions of it — once by catching a contaminated result we had already drafted the conclusions for — which is the only reason the final version is worth believing. Plain markdown, a graph of links, ranked search over normalized text, guarded block-level edits: it turns out that's not just a pleasant way for humans to keep notes. Measured honestly, it's a memory substrate a $4.50 agent can write under mechanical guards and a single retrieval call can read at 96 percent of a hand-built ceiling — one turn, two seconds, a nickel a question — and every page of it opens in your editor.

How this compares

Two judge models appear in this article, both running Mem0's evaluation prompt verbatim, binary CORRECT/WRONG. The frozen test configuration answers and judges with a pinned claude-sonnet-4-6, which re-scores the same calibration answers at 0.70 — the strictest grader on that shared anchor. Every number in this section is sonnet-4-6-judged. That choice makes our numbers comparable and conservative at the same time, and this section is where both properties earn their keep.

The published LOCOMO results — the reason this benchmark exists in the form it does — sit on approximately that scale:

Our row is 652 questions over four test conversations, two of them sealed until first contact; the sealed-only cut scores 0.754 against the 0.752 overall — the check that iterating on previously spent conversations did not inflate the number. Four more test conversations remain unspent. And since our judge is the strictest in the table on the shared calibration, the comparison above should, if anything, understate our rows.

What is comparable across setups is the economics, because tokens and milliseconds don't depend on the judge:

The bottom line we're comfortable defending: under the strictest judge in the table, a $4.50 question-blind curator writing plain markdown, read by one search-and-expand call, matches every published memory system on LOCOMO — and the strongest system on the benchmark is still a competent agent given grep and the raw transcripts, which says as much about the benchmark's scale as about memory systems. The numbers that should move a reader are in the economics table: one turn instead of five or eight, two seconds instead of twelve or twenty, a fraction of the tokens read — for answers that are just as accurate.

The harness, prompts, and per-run records live in the ]