Web Scraping with n8n | Part 3: The AI Web Scraper (One Workflow, Scrape Anything)

Welcome to the finale of our n8n web scraping series!

• In , we tackled the tricky mechanics: pagination loops, infinite scroll, and network capture.

But if you’ve been following along, you know there is still one massive headache in web scraping: "New Site = New Workflow."

Every time you want to scrape a different website, you have to open the browser inspector, hunt for new <div> classes, debug why price_color isn't working, and rewrite your entire flow. It’s exhausting.

Today, we changed that.

In this final part, we are going to build an Automated AI Scraper - a single n8n workflow that can scrape almost anything (Online Stores, Article/News Sites, Job Boards, and more) without you changing a single node.

Whether you are a developer looking to save hours of coding, or a non-technical user who just needs the data without the headache, this tool is designed for you.

💡 TL;DR: Want to start scraping immediately? We have packaged this entire workflow into a ready-to-use template. 👉 . Instead of saying "Find the text inside .product_pod h3 a, we simply send the URL and say product: true. The AI analyzes the visual layout of the page and figures it out, even if the website changes its code tomorrow.

To handle every scenario, we designed three distinct pipelines:

1. 
   

   The "AI Extraction" Pipeline (Automatic)
   
   This is the core of the workflow. You simply select a Category (e.g., E-commerce, Article etc) and a Goal, and the workflow automatically routes you to one of two paths:

   
   
   
   
   
   • 
     Direct Extraction (Fast): If you just need data from the current page (like a "Single Product" or a "Simple List"), the workflow sends a single smart request. No loops, no waiting.
   
   
   • 
     The "Two-Phase" Architecture (For AI crawling): If your goal involves "All Pages" or "Visiting Items," the workflow activates a robust recursive loop:
     
     
     
     
     • 
       Phase 1 (The Crawler): It maps out the URLs you need (looping through pagination or grabbing item links from a list).
     
     
     • 
       Phase 2 (The Scraper): It visits every mapped URL one by one to extract the rich details you asked for.
     
     
     
     
   
   
   
   



2. The "SERP" Pipeline (For SEO Data)
   
   Need search rankings? We included a dedicated path for Search Engine Results Pages (SERP). It uses the specific serp schema to automatically extract organic results, ads, and knowledge panels without you needing to parse complex HTML.




3. The "Manual" Mode (For Raw Control)
   
   Sometimes you don't need AI. We added a "General" path that gives you raw browserHtml, HTTP responses, or Screenshots so you can parse specific data yourself.

Let’s get building.

Step 1: The Control Center

In previous parts, we hardcoded URLs into our nodes. For this tool, that won’t work. We need a flexible User Interface.

1. The Main Interface (Form Trigger)

2. The Brain (Config Generator)

We place a Code Node (the "Zyte Config Generator") to translate your form choices into technical instructions.

• Similar to the single item strategy, but instead of asking for one object, we request a List schema (like productList). The AI identifies the repeating elements on the page and returns them as a clean array.

💡 Design Note: You might notice that the nodes for Strategy 1 and 2 look identical. That is because the heavy lifting (choosing between product vs productList) is actually handled upstream by the Config Generator.

🧑‍💻 Best Practice: In your own production automations, you should usually combine these into a single node to keep your canvas clean. However, for this template, we kept them separate. This makes the logic visually intuitive and allows you to add specific post-processing (like a unique filter) to the List path without accidentally breaking the Single Item path.

C. Details (Current Page):

A hybrid approach. It scans the current list, finds item links, and visits them one by one.

• This enables the pagination loop. The workflow fetches the current page's list, saves the items to a global "Backpack" (memory), detects the "Next Page" link automatically, and loops back to repeat the process until it reaches the end.

E. Crawl Details (All Pages):

The ultimate mode. It crawls all pages (Phase 1) AND visits every single item found (Phase 2).

It is the fastest way to get reliable SERP data for rank tracking or brand monitoring, handling complex layouts automatically.

Step 4: Pipeline 3 – Manual / General Mode

What if sometimes you need to scrape a unique dashboard, a niche directory, or just want to debug the raw HTML yourself. That’s why we included the "Manual" path.

If you select "General / Other" in the form, you are presented with a secondary form offering 5 raw tools:

1. 
   Browser HTML: Returns the full rendered DOM (great for the ).
   


2. 
   Infinite Scroll: Automatically scrolls to the bottom before capturing HTML. ().

This ensures your scraping tool is never useless, even on the most obscure websites.

The Result & Output

Regardless of which pipeline you chose (AI, SERP, or Manual), all data converges at a final Data Collector node.

Get the Workflow

We have packaged this entire logic – the forms, the smart routing, the crawler loops, and the safety checks, into a single template you can import right now from the n8n community.

👉 .

Happy scraping! 🚀🕷️