Someone debugging a test in , which also repaired a batch of one-shot isVisible() reads and matcher-less expect() calls in the E2E suite). To be clear, this is not a story about code-server being sloppy; it is one of the better-maintained projects I have reviewed. That is what makes it a useful opening exhibit. If a leaked .only can sit unnoticed for seven months in a repo like that, it can sit unnoticed anywhere.
The same bug in four more projects
Once I started looking for tests that pass without proving anything, I kept finding them in heavily maintained, widely used codebases. Here is the whole genre in one merged one-line fix, from Carbon Design System ():
expect(likeButton.isDisabled()).toBeTruthy();
isDisabled() returns a Promise, and a Promise is always truthy. Whether the button is disabled, enabled, or not rendered at all, this passes.
Strapi had checks computed and then thrown away (), so the assertion floated off and never participated in the test outcome.
Different-looking bugs, one failure mode: the test runs, goes green, and the green means nothing. I started calling these silent-pass tests. Their dangerous property is camouflage. A missing test at least looks like a gap; a silent-pass test looks like coverage. They are also not rare. In the 100-PR benchmark corpus described below, 33 of the 100 PRs touching Playwright or Cypress specs contained at least one genuine issue of this class.
Check your own suite in the next five minutes
You do not need my tool for the mechanical cases. You need grep. These four are triage: they will surface some legitimate lines (a .toBeDefined() on a config object is fine; on a Locator it is decorative), and each hit takes a few seconds to eyeball. They are written for Playwright; Cypress variants live in the repo.
# 1. Committed focused tests: CI runs one test, silently skips the siblings
grep -rnE "(it|test|describe)\.only\(" e2e/
# 2. Truthiness asserted on a Locator handle: always passes
grep -rnE "expect\([^)]*\)\.(toBeTruthy|toBeDefined|not\.toBeNull)\(" e2e/
# 3. Web-first assertion with no await: floats, never affects the result
grep -rnE "^[[:space:]]*expect\(.+\)\.to(BeVisible|BeHidden|BeEnabled|HaveText|HaveURL|HaveTitle|HaveCount)\(" e2e/
# 4. State reads whose results go nowhere
grep -rnE "^[[:space:]]*(await )?[A-Za-z_.]+\.(isVisible|isEnabled|isDisabled|isHidden)\(\)[[:space:]]*;?[[:space:]]*$" e2e/
Every confirmed hit on #1 or #2 is a test, or a whole suite, that currently proves nothing.
First attempt: just lint it
My first instinct was that this is a lint problem, and some of it genuinely is. expect(locator).toBeTruthy() is a syntactic shape. A leaked .only is about as mechanical as it gets. So I built that layer and published it as two ESLint plugins, , which flag the Carbon and Ghost class of bug on every future PR. (The install one-liner is at the end. The greps above are the no-install version.)
Then I hit the wall. Consider a test that clicks Delete, waits for the modal to close, and ends. Every line is idiomatic. Every assertion that exists is awaited and correct. The problem is the assertion that does not exist: nothing verifies the entity is gone. No AST rule can flag that, because there is no bad node to point at; the defect is the absence of a node. Same for a test named "accepts terms of use" that only asserts the pre-accept state, or an API check like expect([200, 401, 403]).toContain(status) that waves a broken auth endpoint through. Deciding those cases means reading the test the way a reviewer does: what does this test claim to prove, and does the code prove it?
That is a semantic judgment, and it is the kind of work an LLM can do well, provided you constrain it hard.
Measuring it instead of asserting it
"An LLM reviews your tests" is an easy claim to make badly. So before writing this post I benchmarked the approach and published the per-case evidence, so the numbers can be re-derived (). Every one of them fixed a test that was passing in CI while proving too little.
One more thing, because "AI-assisted PRs" have earned a reputation in 2026: these were not sprayed. Each fix was prepared against the target repo's contributing conventions and verified before submission, and the full tally is public — 12 merged, 8 in review. The whole pipeline, including what got dropped and why, lives in the is four Agent Skills for Claude Code, Codex, and other skill-compatible agents, built around a taxonomy of 24 E2E anti-patterns graded P0/P1/P2. P0 means silent always-pass, the tests that cannot fail. P1 means poor diagnostics when they do fail. P2 is maintenance debt.
The centerpiece, e2e-reviewer, runs in two phases. A deterministic scanner (plain bash, regex plus ast-grep, runnable standalone without any LLM) generates candidates; a semantic review phase then verifies each candidate in context and hunts for the absence-shaped defects the scanner cannot see, like the delete test that never checks deletion. That verification step is why the false-positive column reads zero: the LLM's job includes killing bad candidates, not only finding new ones. Alongside it, three skills close the loop. playwright-test-generator writes new coverage held to the same 24-pattern standard — on first run it scaffolds your project's conventions (an AGENTS.md E2E section plus a seed spec) and finishes with a YAGNI audit, so it does not ship a Page Object nobody calls. playwright-debugger and cypress-debugger start from the report files you already have (playwright-report/, mochawesome or JUnit) and classify each failure into one of 15 root-cause categories — flaky timing, selector drift, environment mismatch — each with a concrete fix; hand them a GitHub Actions run and they fetch the artifact themselves. Playwright and Cypress only, deliberately. For anything else, the skills say "out of scope" instead of guessing.
If you take one action from this post, run the greps above. If you want the mechanical patterns caught on every PR from now on, the zero-commitment entry is the lint layer:
npm i -D eslint-plugin-playwright-silent-pass # or eslint-plugin-cypress-silent-pass
The skills, the standalone scanner, and the full benchmark evidence are at · cypress.
The habit transfers even if you never install anything. Next time a suite goes green, ask it: if the feature broke tonight, which of you would actually fail?
SOCIAL SHARE CARD GENERATOR