Your AI writes E2E tests that look right and fail. I built a map so it stops guessing.
AI agents are surprisingly bad at writing end-to-end tests for real apps. Not demo apps - real ones. The kind with zero data-testid attributes, three i18n locales, forms that load their defaults asynchronously, and selectors that only make sense if you wrote the frontend yourself.
I watched agents do this at work: guess a selector, fail, guess again, loop, then finally produce a test that looks completely reasonable and fails on the first run. My favorite failure mode: the agent wrote getByRole('tab', ...) for tabs that are - of course - plain buttons with click handlers. The test was well-written. It was also fiction.
The root cause is simple: the agent has never seen your app. Runtime AI-navigation tools (browser-use, Stagehand, Playwright MCP) explore it live, but they have no persistent memory - every task starts from zero, and exploration burns tokens on every run.
So I built terra-incognita, an open-source "app knowledge compiler". The idea:
A human records navigation sessions in a real browser - you just click through your app once.
A deterministic distiller (no LLM, no API key) compiles those sessions into compact artifacts: an app map, an element dictionary with ranked selectors, a testability report, and a generated skill file.
The agent consumes the map when writing tests: prefer mapped selectors, never invent locators.
It is not a test recorder like codegen, not a runtime agent, not session replay. It compiles human ground truth into knowledge that any agent can read.
Does it help? Depends how big a mess your app is.
I benchmarked it with an A/B harness: same agent, same tasks, same turn budget - with and without the map. Beyond "does the test pass", I scored acceptance (how many acceptance criteria the test actually asserts) and strict (passes AND covers everything), because pass rates alone let narrow, under-asserting tests through.
On a realistically messy production SPA (zero test-ids, three locales, async forms):
- A small model went from 0/8 to 7/8 strict with the map. Without it, its tests looked right and guessed wrong.
- A mid-size model went from 29% to 54% strict; acceptance-criteria coverage rose from 43% to 78%.
- The result I find most interesting: the small model with the map beat a ~7x more expensive model without it on every quality metric. A map instead of a bigger model.
And the honest flip side: on an app that is already well-testable - stable roles, test-ids everywhere - the map produced zero lift. There is nothing to fix. If that is your app, you do not need this tool. Maps are for wild territory.
(These are small pilots - 1-3 runs per task, tasks written by me. Directional evidence, not a published benchmark. A methodology write-up is coming.)
The part I did not expect: honesty needs a linter
The generated skill tells the agent: if a flow is not on the map, say so instead of inventing selectors. Some models follow that rule 100% of the time. Others ignore it routinely and fabricate plausible-looking locators anyway.
Prompt rules are advisory. So the tool ships a mechanical gate: terra lint checks every locator in your test files against the recorded element dictionary and fails the build on anything that is not traceable to a real, observed element. The model's discipline stops mattering; the gate does not care whether it felt like following instructions today.
Try it
The whole pipeline is deterministic and runs offline - record, distill, generate, lint. MIT licensed:
github.com/andrzej-stepien/terra-incognita
terra record http://localhost:3000 --name checkout-flow
terra distill
terra generate-skill --target claude-code
terra lint tests/**/*.spec.ts
If your app is a well-lit city grid, keep walking. If it is terra incognita - give your agent a map and tell me what breaks.