My AI-Generated Tests Kept Passing for the Wrong Reason

I started using AI to generate Playwright test scaffolding a few months ago. Feed it a user flow, whether it's a login form or a multi-step checkout, and you get back a working spec in under a minute that runs and mostly passes on the first try.

I asked a model to write a test for a password reset flow, and it built a spec that filled in the email field, clicked submit, and checked for a success message on screen. That passed every time I ran it, but it never checked whether an email actually went out, whether the link inside it worked, or whether that same link still worked after being used once. The page reported success no matter what the backend did, so the test just checked the page's opinion of itself.

Before I touched it, the spec looked like this:

await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();

All three lines passed, which is exactly the problem: none of them checked anything that actually mattered. Once I rewrote the assertions, it looked like this:

await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
await expect(page.getByText('Check your email')).toBeVisible();

const resetEmail = await getLatestEmail(testUser.email);
const token = extractResetToken(resetEmail.body);

await page.goto(`/reset?token=${token}`);
await page.getByLabel('New password').fill('NewPassword123!');
await page.getByRole('button', { name: 'Reset password' }).click();
await expect(page).toHaveURL('/login');

await page.goto(`/reset?token=${token}`);
await expect(page.getByText('This link has expired')).toBeVisible();

The expired-token check at the end is what caught a real bug: the first version of the reset endpoint let a token be reused as many times as someone wanted, with no expiration after it was used once.

None of this makes AI bad at writing tests. It's good at the boilerplate, at generating locators and shaping a spec file enough that I'm editing instead of starting from nothing, but it can't decide which outcome the business actually depends on, because that context doesn't live in the DOM. A password reset link that never expires looks identical on screen to one that works the way it should, and the same gap shows up anywhere a UI can report success independent of what the backend actually did, checkout confirmations and invite emails included.

I still generate the first draft of most new specs with a model, but I rewrite the assertions myself, against what the feature is actually supposed to guarantee.

Jeff Thoensen is a Context-Driven QA Engineer focused on automation, API testing, and exploratory testing. Find more at jeffthoensen.com.