How to Build a RAG Knowledge Base from Any Documentation Site in 5 Minutes

The Problem

You want to feed documentation into your RAG pipeline, but web scraping gives you a mess of navigation, sidebars, cookie banners, and broken formatting mixed with actual content. You spend hours cleaning up HTML before you can even start building your knowledge base.

The Solution

I built an automated extraction + chunking pipeline that converts any documentation site into clean, structured markdown ready for your vector store.

Step 1: Extract and Chunk the Docs

Using the instead:

{

  "startUrl": "https://docs.python.org/3/library/asyncio.html",

  "maxPages": 1

}

Output is clean markdown with token counts. Good for when you want to control your own chunking strategy or feed single pages into an LLM context window.

How the Cleaning Works

Under the hood, the extractor:

1. 
   Crawls the site using Crawlee (handles rate limiting, dedup, robots.txt)


2. 
   Strips noise — removes <nav>, <footer>, .sidebar, .cookie-banner, <script>, <style>, and 20+ other noise selectors


3. 
   Finds content — looks for <article>, <main>, .markdown-body, .prose, etc.


4. 
   Converts to markdown — preserves headings, code blocks, tables, links, lists


5. 
   Counts tokens — uses cl100k_base encoding for accurate token counts

The result is clean, structured content that's ready for any RAG pipeline.

Links

• 
  — Single-page markdown conversion

Both are open on the Apify Store with pay-per-result pricing. No subscription needed.