You write a scraper with Playwright, wait for the page to load, close the cookie banner, click a filter, and parse a table out of the DOM. Then someone redesigns the page and your selector breaks. The annoying part is that the data probably never lived in the HTML in the first place.
Most modern websites render a UI around structured background requests. The browser loads the shell, runs JavaScript, and calls internal endpoints for prices, availability, inventory, search results, profile data, or whatever the page needs. If you scrape the rendered page, you often process hundreds of kilobytes of layout and tracking code to recover a few kilobytes of JSON.
Look at the network layer before writing browser code
Before reaching for Playwright or Puppeteer, open DevTools and check what the site actually does.
In Chrome:
- Open DevTools
- Go to the Network tab
- Filter by
Fetch/XHR
- Perform the action manually, such as search, filter, paginate, or change dates
- Click the request that returns the data
- Inspect the request URL, method, headers, payload, and response
You will often find something like this:
POST /api/search/hotels HTTP/2
content-type: application/json
x-csrf-token: 8f9c...
{
"checkIn": "2026-03-12",
"checkOut": "2026-03-15",
"city": "Berlin",
"guests": 2
}
And the response is already the thing you wanted:
{
"results": [
{
"id": "hotel_123",
"name": "Example Hotel",
"price": 184,
"currency": "EUR",
"available": true
}
]
}
At that point, scraping the DOM is extra work. You can reproduce the request directly:
curl 'https://example.com/api/search/hotels' \
-X POST \
-H 'content-type: application/json' \
-H 'x-csrf-token: 8f9c...' \
--data '{"checkIn":"2026-03-12","checkOut":"2026-03-15","city":"Berlin","guests":2}'
Or from code:
const res = await fetch('https://example.com/api/search/hotels', {
method: 'POST',
headers: {
'content-type': 'application/json',
'x-csrf-token': process.env.CSRF_TOKEN
},
body: JSON.stringify({
checkIn: '2026-03-12',
checkOut: '2026-03-15',
city: 'Berlin',
guests: 2
})
});
if (!res.ok) {
throw new Error(`Search failed: ${res.status} ${await res.text()}`);
}
const data = await res.json();
console.log(data.results.map(h => [h.name, h.price]));
This is the basic pattern: use the browser to discover the request, not to run every request forever.
is a managed version of this same approach rather than a DOM scraping wrapper.
When a browser is still the right tool
Use browser automation when the browser behavior is the thing you need to test or reproduce.
Good cases for Playwright, Puppeteer, or Selenium:
- End-to-end testing user flows
- Capturing screenshots or PDFs
- Interacting with canvas-heavy or browser-only apps
- Debugging frontend behavior
- Handling flows where the data is only available after complex client-side state changes
- Verifying that the UI actually displays what the API returned
Bad cases:
- Polling prices every five minutes
- Pulling paginated search results
- Checking inventory across many SKUs
- Feeding structured records into a data pipeline
- Giving an agent live availability data
For those, inspect the network requests first. If the data is already JSON, call that layer directly, validate the response shape, and keep the browser out of the hot path unless you actually need it.
SOCIAL SHARE CARD GENERATOR