Most price monitoring systems look simple on paper: point a scraper at a product page, ask an LLM to extract the price, and store the result in a database. The problem arises when the same workflow runs continuously against heavily protected targets such as Amazon and Walmart. At scale, reliability becomes the real challenge. The why comes down to two failures that happen quietly: retrieval and extraction. Retrieval fails when a protected site returns a blank JavaScript shell, incomplete data, and a block page instead of the product data. Extraction fails when the LLM returns a field in a data type or shape that your downstream database write action does not expect. Both failures pass for normal output, so the pipeline keeps running and drops records downstream without raising an error.
This tutorial shows you how you can build a price monitoring agent using ZenRows and Pydantic AI. ZenRows retrieves the page and returns product details with a . The code below shows this failed retrieval directly.
import requests
# Demo page that behaves like a protected target
PROTECTED_URL = "https://www.scrapingcourse.com/antibot-challenge"
resp = requests.get(PROTECTED_URL, timeout=30)
print(f"status code: {resp.status_code}")
print(f"bytes returned: {len(resp.text)}")
print("contains 'price'?", "price" in resp.text.lower())
print(resp.text[:200])
The request returns a 403, so the pipeline does not receive the product page required for price monitoring.
benny@Mac price_monitoring % python3 test.py
status code: 403
bytes returned: 5507
contains 'price'? False
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><meta http-equiv="X-UA-Compatible" content="IE=Edge"><meta nam
benny@Mac price_monitoring %
When this happens, your pipeline would assume that your script has received data and move on. But without any prices, you can't monitor the pages for any product.
2. Extraction returns inconsistent information
Your first run can return data successfully. Across runs, the same field can drift into different formats. For example, one run may return a price with currency symbols attached. Another may return the value as text, and the third as a number. The example below demonstrates how an LLM behaves when this type of data drift occurs.
# Simulated raw LLM extraction responses from two runs
raw_llm_responses = [
{
"product": "Echo Dot (5th Gen)",
"price": 29.99
},
{
"product": "Echo Dot (5th Gen)",
"price": "$29.99"
}
]
def write_to_db(price: float) -> None:
if not isinstance(price, float):
raise TypeError(
f"expected float, got {type(price).__name__}"
)
for i, response in enumerate(raw_llm_responses, start=1):
price = response["price"]
print(
f"run {i}: price={price!r}, "
f"type={type(price).__name__}"
)
for i, response in enumerate(raw_llm_responses, start=1):
try:
write_to_db(response["price"])
print(f"run {i}: write OK")
except TypeError as e:
print(f"run {i}: write FAILED -> {e}")
Below is the output of the extraction script. It returns valid-looking data, but the output schema can drift between runs. Without validation, downstream writes fail when the data type does not match the expected format.
benny@Mac price_monitoring % python3 test_llm.py
run 1: price=29.99, type=float
run 2: price='$29.99', type=str
run 1: write OK
run 2: write FAILED -> expected float, got str
benny@Mac price_monitoring %
Such inconsistencies can break validation logic and downstream workflows, causing pipelines to fail unpredictably during automated or scheduled executions. Next, let's take a look at how ZenRows handles the retrieval problem. However, before we start, let's see everything you need to follow along.
Prerequisites for this tutorial
Use Python 3.10 or newer for this tutorial. If you have an older Python version, you need to create a dedicated virtual environment with a current Python installation to avoid dependency conflicts.
- Install the required packages using
python -m pip install "pydantic-ai-slim[anthropic]" requests python-dotenv. This tutorial was tested locally with Pydantic AI 0.8.1 and Anthropic SDK 0.111.0.
- ZenRows API key. Create an account at handles the retrieval layer for protected pages. It renders the pages, uses proxies, and returns markdown in a single request. You can then pass this output to your extraction agent for the next stage of your pipeline, which you will see in the next section of this piece. The code below shows this flow with
js_renderandpremium_proxy.
CODEimport os
import requests
from dotenv import load_dotenv
load_dotenv()
ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
# Target page (NO markdown formatting)
url = "https://www.scrapingcourse.com/ecommerce/"
params = {
"url": url,
"apikey": ZENROWS_API_KEY,
"js_render": "true",
"premium_proxy": "true",
"proxy_country": "us",
"response_type": "markdown",
}
response = requests.get(
"https://api.zenrows.com/v1/",
params=params,
timeout=60
)
print(f"status code: {response.status_code}")
print(f"bytes returned: {len(response.text)}")
print("\n".join(response.text.splitlines()[:30]))
ZenRows manages anti-bot bypass via
"js_render": "true"which ensures ZenRows loads the page like a real user browser, and"premium_proxy": "true"which enables residential proxies. For this use case, setproxy_country=usto route requests through a US IP address. That keeps our price comparisons consistent across runs. The response type ismarkdown, which is easier for LLMs to parse and read. Your response after running the code should be similar to the output below.
CODEbenny@Mac price_monitoring % python3 zenrows_store.py
status code: 200
bytes returned: 6777
[Skip to navigation](http://www.scrapingcourse.com#site-navigation) [Skip to content](http://www.scrapingcourse.com#content)
[Ecommerce Test Site to Learn Web Scraping](https://www.scrapingcourse.com/ecommerce/)
ScrapingCourse.com
Search for: Search
Menu
- [Shop](https://www.scrapingcourse.com/ecommerce/)
<!--THE END-->
- [Home](https://www.scrapingcourse.com/ecommerce/)
- [Cart](https://www.scrapingcourse.com/ecommerce/cart/)
- [Checkout](https://www.scrapingcourse.com/ecommerce/checkout/)
- [My account](https://www.scrapingcourse.com/ecommerce/my-account/)
<!--THE END-->
- [$0.00 0 items](https://www.scrapingcourse.com/ecommerce/cart/ "View your shopping cart")
- No products in the cart.
# Shop
Default sorting Sort by popularity Sort by latest Sort by price: low to high Sort by price: high to low
Showing 1-16 of 188 results
benny@Mac price_monitoring % ;
Now, let's define your price schema to ensure your LLM reads the extracted content correctly and maps each product field into a predictable format for downstream processing.
Defining the price schema and building the extraction agent
Now that ZenRows has retrieved the data from the website,
Wrapping up
With ZenRows handling retrieval and Pydantic AI validating each extraction, this pipeline remains reliable even when working with protected targets like Amazon and Walmart, which traditional scrapers often fail to scrape or return incomplete data for. ZenRows' strength lies in its ability to reliably access protected sites, with a to build more reliable price-monitoring workflows!
↗ Original-Artikel auf dev.to lesenVollständiger Original-BerichtAusführliche Details, Code-Beispiele & Hersteller-Stellungnahme auf dev.to.
SOCIAL SHARE CARD GENERATOR