We pointed our chaos-QA agent at our own site. It found a shipped bug.

We build an AI QA engineer, so the fair test is the obvious one: point it at

ourselves. On 15 June 2026 we ran Gremlin mode — Prufa's chaos-testing

modality — against our own marketing site, prufa.dev. It found a real,

user-facing bug that our CI had gone green on and shipped that same day. Here is

the whole run, including the parts where the tool was wrong about itself.

What Gremlin mode actually does

A normal Prufa flow checks a path you already know to check. Gremlin is for the

paths you didn't. An LLM-backed agent drives a real browser as a deliberately

difficult user — a confused newbie, an impatient double-clicker, a fat-finger

typist, a back-button masher, a hostile poker — and chooses its own next action

every step. It is the part of QA that needs a model: absorbing an unfamiliar UI

and deciding what a frustrated human would try next.

What the agent never does is decide whether anything broke. That is the same

invariant as the rest of Prufa —

the LLM navigates, plain code verifies —

and it is the whole reason a finding from an LLM-driven tester can be trusted: a

separate layer of deterministic detectors grades the run. A 500 response, an

uncaught exception, a form that accepts invalid input, content wider than the

viewport, two clickable elements overlapping — those are facts, read off the

live page, not opinions.

The bug: a mobile overflow CI had just shipped

Across three personas, every run reported the same verified finding at the

390px mobile viewport: the page was 103 pixels wider than the screen, with

the "Run a free audit" button in the header hanging off the right edge.

Here is the part that makes the case for chaos QA. Earlier that same day, a

commit titled "fix" had added exactly the rule meant to prevent this:

@media (max-width: 520px) { .header-cta { display: none; } }

It never applied. The button is styled by a.btn-primary { display: inline-block },

whose selector specificity (0,1,1) outranks the bare .header-cta (0,1,0), so

the display: none was silently overridden on every phone-width render. The CSS

was valid. The build passed. The linter was happy. CI was green. And the bug

shipped to production, where it sat 103px wide until an agent that had never seen

our codebase resized the viewport and measured the document.

The fix was to out-specify the button:

@media (max-width: 520px) { header a.header-cta { display: none; } }

header a.header-cta is specificity (0,1,2), which beats a.btn-primary

regardless of source order. After the change, a fresh build measured 0px of

horizontal overflow at 390px and the button correctly hidden. The class of bug

matters here: nothing errored. A test that asserts known selectors would have

stayed green forever, because the breakage was in a layout dimension no one had

written an assertion about. You catch that by measuring the rendered page, not by

re-running the path you already trusted.

The safety guarantee, demonstrated on a live site

A chaos tester loose on a real site is only acceptable if it cannot change

anything. In Prufa, mutations are denied by default: the run is dry-run and a

network-layer guard aborts every non-GET request before it leaves the browser. A

destructive click becomes a "would have mutated" finding instead of an action.

We didn't have to take that on faith — the run logged it. Across the three

personas the agent attempted between 0 and 4 mutations each; every one was

blocked, and the run recorded which control it would have submitted. Real

payment instruments are never used at all. To let Gremlin submit forms for real,

you explicitly authorise a domain you own — and even then, hard caps bound how

many submissions it can make.

Where the tool was wrong about itself

The honest part. In an earlier run, two of the gremlin's own detectors fired

on things that were not bugs:

• A "dead-end / error page" detector matched the bare string 500 in ordinary
  marketing copy (think "save $500"), calling a healthy page an error page.


• A "bad input accepted" detector treated any navigation after a form fill as a
  successful submission — so clicking a normal link after typing in a field
  looked like the app had swallowed invalid input.

A verified finding that turns out to be noise costs more trust than a missed bug

costs coverage, so we did not ship around it. We added a detector

false-positive policy: the error-page check now requires a strong error phrase

in the page's prominent text (title or heading) on an error-shaped page, not a

substring match anywhere in the body; the bad-input check now requires a real

form submission — an actual non-GET request — before it fires. Both false

positives are gone, and the genuine findings (the mobile overflow) still land.

We also measured discovery quality directly. On a seeded-bug fixture with five

planted bugs, the agent's first pass found four of five (0.80 coverage); after

we gave it an exploration frontier — a running list of same-site pages it

hasn't visited yet, fed back into each decision — it found all five (1.00),

because it stopped looping one corner and started covering the whole app. That

number is fixture discovery quality, not a claim about your site; the point is

that "does the chaos actually find the planted bugs" is something we test, with

a number, not assert.

Why we publish the misses

A QA product that only tells flattering stories about itself is exactly the

product you shouldn't trust to test you. The mobile bug is a good demo. The

false positives are a better one: they show the failure mode that matters for an

LLM-driven tester — a confident, wrong "this is broken" — and they show the line

we hold against it. The model proposes; plain code disposes; and when plain code

gets it wrong, we fix the plain code, in the open.

Gremlin mode is available on any paid plan — read how it works on

the chaos-testing page, or

run a free audit to see the deterministic side of the same engine on your

own URL first.