Building the Ultimate Reddit Scraper: A Full-Featured, API-Free Data Collection Suite
December 2024 | By Sanjeev Kumar
TL;DR
I built a complete Reddit scraper suite that requires zero API keys. It comes with a beautiful Streamlit dashboard, REST API for integration with tools like Grafana and Metabase, plugin system for post-processing, scheduled scraping, notifications, and much more. Best of all—it’s completely open source.
🔗 GitHub: reddit-universal-scraper
The Problem
If you’ve ever tried to scrape Reddit data for analysis, research, or just personal projects, you know the pain:
- Reddit’s API is heavily rate-limited (especially after the 2023 API changes)
- API keys require approval and are increasingly restricted
- Existing scrapers are often single-purpose - scrape posts OR comments, not both
- No easy way to visualize or analyze the data after scraping
- Running scrapes manually is tedious - you want automation
I decided to solve all of these problems at once.
________________________________________
The Solution: Universal Reddit Scraper Suite
After weeks of development, I created a full-featured scraper that:
Feature What It Does
📊 Full Scraping Posts, comments, images, videos, galleries—everything
🚫 No API Keys Uses Reddit’s public JSON endpoints and mirrors
📈 Web Dashboard Beautiful 7-tab Streamlit UI for analysis
🚀 REST API Connect Metabase, Grafana, DuckDB, and more
🔌 Plugin System Extensible post-processing (sentiment analysis, deduplication, keywords)
📅 Scheduled Scraping Cron-style automation
📧 Notifications Discord & Telegram alerts when scrapes complete
🐳 Docker Ready One command to deploy anywhere
________________________________________
Architecture Deep Dive
How It Works Without API Keys
The secret sauce is in the approach. Instead of using Reddit’s official (and restricted) API, I leverage: - Reddit’s public JSON endpoints: Every Reddit page has a .json suffix that returns structured data
- Multiple mirror fallbacks: When one source is rate-limited, the scraper automatically rotates through alternatives like Redlib instances
- Smart rate limiting: Built-in delays and cool-down periods to stay under the radar
MIRRORS = [
"",
"",
""
]
When one source fails, it automatically tries the next. No manual intervention needed.
The Core Scraping Engine
The scraper operates in three modes: - Full Mode - The complete package
python main.py python --mode full --limit 100
This scrapes posts, downloads all media (images, videos, galleries), and fetches comments with their full thread hierarchy. - History Mode - Fast metadata-only
python main.py python --mode history --limit 500
Perfect for quickly building a dataset of post metadata without the overhead of media downloads.
Monitor Mode - Live watching
python main.py python --mode monitor
Continuously checks for new posts every 5 minutes. Ideal for tracking breaking news or trending discussions.
The Dashboard Experience
One of the standout features is the 7-tab Streamlit dashboard that makes data exploration a joy:
📊 Overview Tab
At a glance, see: - Total posts and comments - Cumulative score across all posts - Media post breakdown - Posts-over-time chart - Top 10 posts by score
📈 Analytics Tab
This is where it gets interesting: - Sentiment Analysis: Run VADER-based sentiment scoring on your entire dataset - Keyword Cloud: See the most frequently used terms - Best Posting Times: Data-driven insights on when posts get the most engagement
🔍 Search Tab
Full-text search across all scraped data with filters for: - Minimum score - Post type (text, image, video, gallery, link) - Author - Custom sorting
💬 Comments Analysis
• View top-scoring comments
• See who the most active commenters are
• Track comment patterns over time
⚙️ Scraper Controls
Start new scrapes right from the dashboard! Configure: - Target subreddit/user - Post limits - Mode (full/history) - Media and comment toggles
📋 Job History
Full observability into every scrape job: - Status tracking (running, completed, failed) - Duration metrics - Post/comment/media counts - Error logging
🔌 Integrations
Pre-configured instructions for connecting: - Metabase - Grafana - DreamFactory - DuckDB
The Plugin Architecture
I designed a plugin system to allow extensible post-processing. The architecture is simple but powerful:
class Plugin:
"""Base class for all plugins."""
name = "base"
description = "Base plugin"
enabled = True
def process_posts(self, posts):
return posts
def process_comments(self, comments):
return comments
Built-in Plugins
Sentiment Tagger Analyzes the emotional tone of every post and comment using VADER sentiment analysis:
class SentimentTagger(Plugin):
name = "sentiment_tagger"
description = "Adds sentiment scores and labels to posts"
def process_posts(self, posts):
for post in posts:
text = f"{post.get('title', '')} {post.get('selftext', '')}"
score, label = analyze_sentiment(text)
post['sentiment_score'] = score
post['sentiment_label'] = label
return posts
Deduplicator Removes duplicate posts that may appear across multiple scraping sessions.
Keyword Extractor Pulls out the most significant terms from your scraped content for trend analysis.
Creating Your Own Plugin
Drop a new Python file in the plugins/ directory:
from plugins import Plugin
class MyCustomPlugin(Plugin):
name = "my_plugin"
description = "Does something cool"
enabled = True
def process_posts(self, posts):
# Your logic here
return posts
Enable plugins during scraping:
python main.py python --mode full --plugins
REST API for External Integrations
The REST API opens up the scraper to a whole ecosystem of tools:
python main.py --api
API at
Key Endpoints
Endpoint Description
GET /posts List posts with filters (subreddit, limit, offset)
GET /comments List comments
GET /subreddits All scraped subreddits
GET /jobs Job history
GET /query?sql=... Raw SQL queries for power users
GET /grafana/query Grafana-compatible time-series data
Real-World Integration: Grafana Dashboard
- Install the “JSON API” or “Infinity” plugin in Grafana
- Add datasource pointing to ."
export TELEGRAM_BOT_TOKEN="123456:ABC..."
export TELEGRAM_CHAT_ID="987654321"
Now you get notified with scrape summaries directly in your preferred platform.
Dry Run Mode: Test Before You Commit
One of my favorite features is dry run mode. It simulates the entire scrape without saving any data:
python main.py python --mode full --limit 50 --dry-run
Output:
🧪 DRY RUN MODE - No data will be saved
🧪 DRY RUN COMPLETE!
📊 Would scrape: 100 posts
💬 Would scrape: 245 comments
Perfect for: - Testing your scrape configuration - Estimating data volume before committing - Debugging without cluttering your dataset
Docker Deployment
Quick Start
Build
docker build -t reddit-scraper .
Run a scrape
docker run -v ./data:/app/data reddit-scraper python --limit 100
Run with plugins
docker run -v ./data:/app/data reddit-scraper python --plugins
Full Stack with Docker Compose
docker-compose up -d
This spins up: - Dashboard at
Deploy to Any VPS
ssh user@your-server-ip
git clone
cd reddit-universal-scraper
Install dependencies
pip install -r requirements.txt
Your first scrape
python main.py python --mode full --limit 50
Launch the dashboard
python main.py --dashboard
That’s it! You’re now scraping Reddit like a pro.
Contributing
This is an open-source project and contributions are welcome! Whether it’s: - Bug fixes - New plugins - Documentation improvements - Feature suggestions
Open an issue or submit a PR on GitHub.
If you found this useful, consider giving the project a ⭐ on GitHub!
Connect
• GitHub: @ksanjeev284
• Project: reddit-universal-scraper
SOCIAL SHARE CARD GENERATOR