Why I Gave Up on Regex and Built an AI Data Extractor

I’ve been scraping the web for years. It’s a love-hate relationship: the thrill of finally pulling the data you need, followed by the despair when the site redesigns and everything breaks. Last month, I hit a wall. I needed to extract product specs from dozens of e-commerce pages. Each page had the same data (name, price, description, dimensions) but the HTML structure varied wildly. Some used <dl>, some <table>, some just <div> soup with inline CSS. My trusty regex and BeautifulSoup pipeline turned into a nightmare of conditional branches.

The regex abyss

I started optimistically. Write a few patterns, test, repeat. But soon my code looked like this:

import re

from bs4 import BeautifulSoup



def extract_price(html):

    patterns = [

        r'\$?\d+[\.\,]?\d*',

        r'price[\s:]*(\$?\d+[\.\,]?\d*)',

        r'<span class="price">(.*?)<\/span>',

        # ... more patterns

    ]

    for pat in patterns:

        match = re.search(pat, html, re.I)

        if match:

            return match.group(1) if match.lastindex else match.group()

    return None

It worked… for about three pages. Then a new site used € instead of $, or had the price embedded in a JavaScript object. I’d add more patterns. Then another site used an image of the price. I cried a little.

The BeautifulSoup maze

I tried to be smarter: parse the HTML structure. But every site had a unique layout. I wrote a function that tried all common selectors:

def find_price(soup):

    for selector in [

        '.price', '.product-price', '#price',

        '[itemprop="price"]', 'meta[name="price"]',

    ]:

        el = soup.select_one(selector)

        if el:

            return el.get('content') or el.text.strip()

    return None

Good, but still brittle. One site used .prc, another put price inside a <s> tag that was actually the old price. The false positives mounted. I needed a different approach.

The lightbulb: treat it like a natural language understanding problem

I realized that what I really wanted was to read the page like a human: ignore the markup and just understand the semantic meaning. That’s exactly what large language models (LLMs) are good at. Why fight the HTML when I could ask an AI to extract the data?

The idea: feed the raw HTML (or a cleaned text version) to an LLM with a prompt that says "Give me the price, name, and description in JSON." The model can handle variations because it understands context.

The approach: structured extraction with LLMs

I chose to use LangChain with OpenAI’s GPT-4 (but later found cheaper alternatives). Here’s the core idea:

1. Fetch the HTML.


2. Strip script/style tags and reduce noise (optional, but helps with cost).


3. Send the text + a prompt to the LLM, requesting a JSON response.


4. Parse the JSON.

Example code

import requests

from bs4 import BeautifulSoup

from langchain.chat_models import ChatOpenAI

from langchain.schema import HumanMessage, SystemMessage



# Your OpenAI API key (or any LLM provider)

llm = ChatOpenAI(model="gpt-4", temperature=0)



def extract_product_info(url):

    # Fetch and clean HTML

    resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"})

    soup = BeautifulSoup(resp.text, "html.parser")

    # Remove script/style tags

    for tag in soup(["script", "style", "nav", "footer"]):

        tag.decompose()

    text = soup.get_text(separator=" ", strip=True)[:4000]  # limit tokens



    system_prompt = "You extract product information from unstructured text. Respond only with a JSON object containing: name, price, description, dimensions (if present)."

    user_prompt = f"Extract data from this text:\n\n{text}"



    response = llm([

        SystemMessage(content=system_prompt),

        HumanMessage(content=user_prompt)

    ])



    # Try to parse JSON (handle potential formatting)

    import json

    try:

        result = json.loads(response.content)

    except:

        # Fallback: extract JSON from markdown code blocks

        import re

        match = re.search(r'```



(?:json)?\s*([\s\S]*?)



```', response.content)

        if match:

            result = json.loads(match.group(1))

        else:

            raise

    return result



# Example usage

url = "https://example.com/product/123"

info = extract_product_info(url)

print(info)

# {'name': 'Widget Pro', 'price': '$29.99', 'description': 'A durable widget...', 'dimensions': '10x5x3 cm'}

Cost and speed trade-offs

This approach isn’t free. A request to GPT-4 costs around $0.03–$0.10 depending on input size. For a hundred products, that’s $3–10. Speed is also slower (2–5 seconds per page). I mitigated by:

• Using GPT-3.5-turbo for simpler pages (much cheaper, about $0.001 per call).


• Reducing input size: only send the visible text around the product area (use XPath or CSS to extract main content).


• Batching: if multiple items are on one page, ask for all in one call.

When it fails

LLMs aren’t perfect. I’ve seen hallucinations: inventing a price when none exists, or mixing up name and description. To guard, I always validate the output against expected types and ranges (e.g., price should match \d+\.\d{2}). Also, I set temperature=0 to reduce randomness.

Another limitation: if the page is mostly JavaScript-rendered, you need a headless browser first. That adds complexity.

Alternatives I considered

• 
  Commercial APIs like ai.interwestinfo.com (I haven't used it personally, but it offers a similar service). The advantage is no need to manage API keys or prompts; the downside is vendor lock-in and potentially higher per-request costs.


• 
  Local models (LLaMA, Mistral) via Ollama: free but slower and less accurate for extraction.


• 
  Fine-tuning: overkill for a one-off project, but could be worth it for a recurring domain.

What I learned

• 
  Don’t fight the format. If the data is unstructured, use a model that understands language, not markup.


• 
  LLM extraction is a complement, not a replacement. For well-structured pages, traditional parsing is faster and cheaper.


• 
  Prompt engineering matters a lot. A poorly written prompt returns garbage. Experiment and iterate.

What I’d do differently next time

I’d start with a small test suite of 5–10 representative pages and evaluate accuracy before scaling. I’d also use a more structured output format: LangChain offers PydanticOutputParser that enforces schema. That would catch hallucinations early.

Closing thoughts

Regex and BeautifulSoup are still my go-tos for stable APIs or consistent HTML. But when the chaos level goes beyond 7/10, I now reach for an AI model. It’s like having a junior developer who can read any page—just a bit slower and more expensive.

What’s your approach for dealing with wildly variable web pages? Do you stick with pattern matching or have you tried AI extraction?