Introduction
End-to-end tests are great for ensuring application reliability, but they can bring maintenance headaches. Even minor UI changes can break tests, leaving developers and QAs wasting hours debugging.
In this article, I’ll show you how to leverage ChatGPT or Copilot to fix Playwright tests automatically. You’ll learn how to pre-generate an AI prompt for any failing test and attach it to the HTML report. That way, you can easily copy and paste the prompt into AI tools and instantly get suggestions for fixing the test.
Let’s dive in!
Plan
The solution boils down to three steps:
- Detect when a Playwright test fails
- Generate an AI prompt with relevant context:
- Error message
- Test code snippet
- ARIA snapshot of the page
- Error message
- Attach the prompt to the Playwright HTML report
Step 1: Detecting a Failed Test
Detecting a failed test in Playwright can be done in a function that removes these special symbols:
const clearedErrorMessage = stripAnsiEscapes(testInfo.error.message);
Cleared error message:
TimeoutError: locator.click: Timeout 1000ms exceeded.
Call log:
- waiting for getByRole('button', { name: 'Get started' })
This cleaned-up message can be inserted into the prompt template.
Code Snippet
The test code snippet is crucial for AI to generate the necessary code changes. Playwright often includes these snippets in its reports, for example:
4 | test('get started link', async ({ page }) => {
5 | await page.goto('https://playwright.dev');
> 6 | await page.getByRole('button', { name: 'Get started' }).click();
| ^
7 | await expect(page.getByRole('heading', { level: 3, name: 'Installation' })).toBeVisible();
8 | });
You can see how Playwright internally
ARIA Snapshot
, which you can call on any element. For AI to fix a test, it makes sense to include the ARIA snapshot of the entire page, retrieved from the root <html> element:
const ariaSnapshot = await page.locator('html').ariaSnapshot();
Assembling the Prompt
Finally, combine all the pieces into one prompt:
const errorMessage = stripAnsiEscapes(testInfo.error.message);
const snippet = getCodeSnippet(testInfo.error);
const ariaSnapshot = await page.locator('html').ariaSnapshot();
const prompt = promptTemplate
.replace('{title}', testInfo.title)
.replace('{error}', errorMessage)
.replace('{snippet}', snippet)
.replace('{ariaSnapshot}', ariaSnapshot);
Example of the generated prompt:
Fix the error in the Playwright test "get started link".
TimeoutError: locator.click: Timeout 1000ms exceeded.
Call log:
- waiting for getByRole('button', { name: 'Get started' })
Code snippet of the failing test:
test('get started link', async ({ page }) => {
await page.goto('https://playwright.dev');
await page.getByRole('button', { name: 'Get started' }).click();
await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});
ARIA snapshot of the page:
- document:
- region "Skip to main content":
- link "Skip to main content"
- navigation "Main":
- link "Playwright logo Playwright":
- img "Playwright logo"
- text: Playwright
...
Step 3: Attach the Prompt to the Report
When the prompt is built, you can attach it to the test using testInfo.attach:
await testInfo.attach('🤖 Fix with AI', { body: prompt });
Now, whenever a test fails, the HTML report will include an attachment labeled “🤖 Fix with AI.”
Testing
To try out the "Fix with AI" prompt, I created a simple test to validate the Get started link on the Playwright
HTML report with AI prompt attachment
You can expand the attachment and copy the prompt by clicking the small button in the top-right corner:
ChatGPT output for fixing the test
ChatGPT correctly identifies that the button role is incorrect and recommends using a link role. After applying the suggestion, the test passes! 🎉
Improving the Prompt
Although ChatGPT gave a detailed explanation, in day-to-day workflows, you might prefer a more concise output that focuses on code changes. I've made many experiments and arrived at this prompt template:
You are an expert in Playwright testing.
Fix the error in the Playwright test "{title}".
- Provide response as a diff highlighted code snippet.
- Strictly rely on the ARIA snapshot of the page.
- Avoid adding any new code.
- Avoid adding comments to the code.
- Avoid changing the test logic.
- Use only role-based locators: getByRole, getByLabel, etc.
- For 'heading' role try to adjust the level first.
- Add a concise note about applied changes.
- If the test may be correct and there is a bug in the page, note it.
{error}
Code snippet of the failing test:
{snippet}
ARIA snapshot of the page:
{ariaSnapshot}
With this refined prompt, ChatGPT usually provides a succinct fix. You simply copy the suggested code and paste it back into your test:
ChatGPT fixes link text
It’s important to distinguish actual bugs from legitimate UI text changes. That’s why I prefer to see the code diff first and analyze what’s happening.
Check 3: Remove Link Name
What if the locator matches multiple elements on the page? Let’s test if AI can identify the correct one.
I remove the link’s name property, causing the locator to match all links on the page:
test('get started link', async ({ page }) => {
await page.goto('https://playwright.dev');
- await page.getByRole('link', { name: 'Get started' }).click();
+ await page.getByRole('link').click();
await expect(page.getByRole('heading', { name: 'Installation' })).toBeVisible();
});
The test fails with:
Error: locator.click: Error: strict mode violation:
getByRole('link') resolved to 39 elements:
ChatGPT’s suggestion:
Potential place for "Fix with AI" button in Playwright VS Code extension
2. HTML Report Enhancements
Similarly, a button in the Playwright HTML report could automatically send a request to a configured AI model and display a suggested fix. This would be especially helpful for team members who focus on reports and don't use IDE.
Here’s a potential spot for a “Fix with AI” button in the report:
To integrate the “Fix with AI” flow into your own project, follow these steps:
- Ensure you’re on Playwright 1.49 or newer
- Copy the
fix-with-ai.tsfile into your test directory
Register the AI-attachment fixture:
CODEimport { test as base } from '@playwright/test';
import { attachFixWithAI } from './fix-with-ai';
export const test = base.extend<{ fixWithAI: void }>({
fixWithAI: [async ({ page }, use, testInfo) => {
await use();
await attachFixWithAI(page, testInfo);
}, { scope: 'test', auto: true }],
});
Run your tests and open the HTML report to see the “Fix with AI” attachment under any failed test
From there, simply copy and paste the prompt into ChatGPT or GitHub Copilot, or use Copilot’s edits mode to automatically apply the code changes.
I’d love to hear your thoughts or prompt suggestions for making the “Fix with AI” process even more seamless. Feel free to share your feedback in the comments.
Thanks for reading, and happy testing with AI! ❤️
SOCIAL SHARE CARD GENERATOR