How I Stopped Regexing HTML Tables and Started Using AI for Data Extraction

I've been scraping data from the web for years. You'd think I'd have learned by now: never use regex on HTML. But sometimes, when you're staring at a messy table with inconsistent classes, random whitespace, and nested elements that barely qualify as valid markup, the temptation to just throw a regex at it is overwhelming.

I found myself in that exact situation last month. I needed to extract property listings from a dozen different real estate websites. Each site had its own quirks. One used <table> tags with rowspan and colspan that made BeautifulSoup cry. Another had dynamic content loaded via JavaScript that my initial scraping setup couldn't even see.

This is the story of how I gave up on perfect parsing and let an AI handle the messy middle.

The Problem: Fragile Parsers

My first attempt was the classic approach: Python + requests + BeautifulSoup. For sites with clean semantic HTML, this worked beautifully. But the real world is full of edge cases:

• Missing closing tags


• Inline styles overriding table structure


• Random <br> elements splitting text across rows


• Data that spans multiple cells visually but not in the DOM

I wrote custom functions for each site. They worked for a week. Then the site updated its layout, and my parser broke. Again.

I tried regex as a last resort (I know, I know). Here’s a snippet of the mess I ended up with:

import re



def extract_price_from_html(html):

    pattern = r'<span[^>]*class="price[^"]*"[^>]*>([0-9,.$]+)</span>'

    match = re.search(pattern, html)

    return match.group(1) if match else None

This worked for exactly one site, on a good day, with perfect formatting. Any minor change broke it. I was maintaining a fragile house of cards.

What Didn't Work: More Rules

My next idea was to use lxml with XPath. More precise, but still brittle. I even tried building a custom state machine to track table cell positions — overengineering at its finest. I spent two days writing code that handled 80% of cases, then gave up on the long tail.

I needed something that could understand the meaning of the data, not just its layout.

What Eventually Worked: An AI-Based Extraction Layer

I started experimenting with large language models (LLMs) to parse the HTML text directly. The idea: dump the raw HTML (or a cleaned version) into an AI API and ask it to return structured JSON. No parsing rules, no regex, no XPath — just a prompt.

I found a service that abstracts this into a simple REST endpoint. The core insight is that instead of writing code to find the price in a table, you tell the AI what the price looks like and let it figure out the context.

Here's the approach I settled on:

1. Fetch the page HTML.


2. Extract the main content area (strip out headers, footers, scripts).


3. Send that content to the AI API with a prompt describing the desired output schema.

I built a small Python class around it:

import requests

import json

from bs4 import BeautifulSoup



class AIDataExtractor:

    def __init__(self, api_key):

        self.api_key = api_key

        self.base_url = "https://ai.interwestinfo.com/extract"  # Example endpoint



    def extract_listings(self, html, schema):

        # Preprocess: extract only the visible text table-ish parts

        soup = BeautifulSoup(html, 'html.parser')

        # Remove scripts, styles, nav

        for tag in soup(['script', 'style', 'nav', 'header', 'footer']):

            tag.decompose()

        clean_text = soup.get_text(separator='\n', strip=True)



        prompt = f"""Extract property listings from the following web page content.

Return a JSON array of objects with these fields: address, price, bedrooms, bathrooms, square_feet.

If a field is not found, set it to null.



Content:

{clean_text[:5000]}  # limit to avoid token overflow

"""



        response = requests.post(

            self.base_url,

            json={"prompt": prompt, "model": "default"},

            headers={"Authorization": f"Bearer {self.api_key}"}

        )

        response.raise_for_status()

        return response.json()

The actual API call returns a JSON object with a listings key. I then validate and transform it into my data model.

Code Example in Action

Let's say I'm scraping a real estate site. Here's how I'd use the extractor:

import requests



url = "https://example-realty.com/listings"

page_html = requests.get(url).text



extractor = AIDataExtractor(api_key="sk-...")

listings = extractor.extract_listings(page_html, schema)



for listing in listings:

    print(f"{listing['address']} - {listing['price']}")

The first few results were surprisingly accurate — about 90% of fields populated correctly. The errors were mostly on edge cases like “Price Upon Request” or missing square footage. I added a second pass of validation: check that price is numeric, address exists, etc. If a field is null, I can either skip that listing or use a fallback parser.

Lessons Learned / Trade-offs

This approach isn't a silver bullet. Here's what I discovered:

• 
  Cost: AI API calls cost money, especially for high-volume scraping. Each request might be $0.01–$0.05 depending on the model. For 10,000 listings, that adds up.


• 
  Latency: Calling an API is slower than local parsing. Each request takes 1–3 seconds. If you're scraping thousands of pages, this can take hours.


• 
  Hallucinations: The AI sometimes invents data if it can't find it. For example, if a price is missing, it might guess “$500,000” from context or just make something up. You absolutely need validation steps.


• 
  Rate Limits: Many AI APIs have strict rate limits. You'll need to throttle or rotate accounts.


• 
  Consistency: The same page can return slightly different JSON each time due to model non-determinism. Not ideal for production ETL pipelines.

When not to use this:

• You have well-structured HTML that BeautifulSoup can handle (e.g., government data tables).


• You're scraping billions of pages (cost kills you).


• You need perfect, deterministic output every time.

The sweet spot is when you have a moderate volume of pages with inconsistent structure, and you can afford a few cents per page to avoid writing custom parsers.

What I'd Do Differently Next Time

I'd start with the AI approach from day one, but I'd also build a caching layer to avoid re-requesting the same page. I'd also use a smaller, cheaper model for simple extractions and reserve the powerful (expensive) models for truly messy pages. Some services let you specify model size in the request.

I'd also mix approaches: use regex for high-confidence fields (like prices prefixed with '$') and the AI as a fallback for the long tail. A hybrid pipeline would reduce costs while still handling the weird edge cases.

The Bigger Lesson

AI isn't just about generating text or images. It's a tool for understanding context — something that's incredibly hard to do with deterministic code. For data extraction, it turns the problem from “write a parser for every layout” into “describe what you want in English and let the model figure it out.” That trade-off is worth it for many real-world scraping projects.

What's your go-to approach when you hit a wall with structured data extraction? Do you roll your own parser, or have you tried the AI route? I'd love to hear about your experiences – especially the horror stories.