reddit-universal-scraper

🤖 Universal Reddit Scraper Suite

A full-featured Reddit scraper with analytics dashboard, REST API, scheduled scraping, plugins, and more. Two collection modes: free browser-page capture, or approved OAuth API access.

✨ Features

Feature	Description
📊 Full Scraping	Available posts, loaded comments, images, videos, galleries
📈 Web Dashboard	Beautiful Streamlit UI with 7 tabs
🚀 REST API	Connect Metabase, Grafana, DuckDB
🔌 Plugin System	Extensible post-processing (sentiment, dedupe, keywords)
📋 Job Tracking	Full history with status, duration, errors
🧪 Dry Run Mode	Test scrape rules without saving data
📦 Parquet Export	Analytics-ready format for DuckDB/warehouses
😀 Sentiment Analysis	Analyze post/comment sentiment
📅 Scheduled Scraping	Cron-style job scheduling
📧 Notifications	Discord & Telegram alerts
🗄️ SQLite Database	Structured storage with auto-backup

---

🚀 Quick Start

# Install dependencies
pip install -r requirements.txt

# Configure approved API credentials (see .env.example)
# Copy .env.example to .env only if you do not already have .env
python main.py --check-access

# Scrape a subreddit
python main.py python --mode full --limit 100

# Launch dashboard
python main.py --dashboard
# Opens at http://localhost:8501

📋 Requirements

• Python 3.10+
• ffmpeg (optional, for video with audio)

# Windows (via chocolatey)
choco install ffmpeg

# macOS
brew install ffmpeg

# Ubuntu/Debian
sudo apt install ffmpeg

---

Free browser extension

The Browser Collector Companion works independently in Chrome/Edge while you browse Reddit signed in. It captures loaded posts, comments and media links, supports filters and session limits, and exports JSON or CSV without paid services or API keys. It can also send collections to the application for database storage, dashboard analytics and search. Collect only data you are authorized to use.

Load the extension/ folder through Developer mode → Load unpacked, reload Reddit, then use Start capture or Capture this page. It observes loaded content as you browse; it does not retrieve an entire archive or automatically expand unloaded comments.

# Connect the extension to the application
python main.py --api
# In another terminal: copy this pairing token into the extension
python main.py --bridge-token

# Or import a standalone JSON export without starting the API
python main.py --import-browser path/to/export.json

# Build the unpackable release ZIP
python scripts/package_extension.py

Pairing uses optional localhost permissions and a local token. The browser's Reddit cookies remain in the browser. Python proxy support remains available, including stable country/session targeting; the extension uses your existing browser proxy settings.

Reddit access and reliability

Reddit blocks unauthenticated API traffic. The direct Python scraper uses oauth.reddit.com with a descriptive User-Agent, cached OAuth tokens, token renewal, bounded retries and shared request pacing. Both sync and async scrapers respect Retry-After and Reddit rate-limit headers. API requests are serialized within each client; media downloads remain concurrent. Independent processes share your app quota, so schedule them accordingly.

See Reddit access setup for credentials, restrictions, troubleshooting and the Developer Platform migration timeline. Missing credentials, denied access, HTML responses and exhausted retries produce explicit failures. CLI failures exit with status 1; jobs preserve collected posts and report failed status when a later request fails.

An optional stable network proxy can be configured with PROXY_URL or --proxy. ScrapingAnt country/session targeting is available through PROXY_COUNTRY / --proxy-country and PROXY_SESSION_ID / --proxy-session. --proxy none disables it. Mirrors, browser impersonation, warm-up requests and automatic proxy rotation are no longer used. A proxy does not grant API approval.

Listings expose a limited window of posts, not a complete historical archive. Comment collection includes the initially loaded tree up to the configured depth; more placeholders are not expanded. Deleted or restricted content cannot be recovered.

---

📖 All Commands

🔄 Scraping

# Collect available posts + media + loaded comments
python main.py delhi --mode full --limit 100

# Fast history-only (no media/comments)
python main.py delhi --mode history --limit 500

# Live monitor (checks every 5 min)
python main.py delhi --mode monitor

# Scrape a user's posts
python main.py spez --user --mode full --limit 50

# Skip media or comments
python main.py delhi --no-media --limit 200
python main.py delhi --no-comments --limit 200

🧪 Dry Run Mode

Test scrape rules without saving any data:

python main.py python --mode full --limit 50 --dry-run

Output:

🧪 DRY RUN MODE - No data will be saved
🧪 DRY RUN COMPLETE!
   📊 Would scrape: 100 posts
   💬 Would scrape: 245 comments

🔌 Plugins

Enable post-processing plugins:

# List available plugins
python main.py --list-plugins

# Run with plugins enabled
python main.py python --mode full --plugins

Built-in Plugins:

Plugin	Description
sentiment_tagger	Adds sentiment scores to posts
deduplicator	Removes duplicate posts
keyword_extractor	Extracts top keywords

Create custom plugins in plugins/ folder.

📊 Dashboard

python main.py --dashboard
# Opens at http://localhost:8501

Dashboard Tabs:

• 📊 Overview - Stats & charts
• 📈 Analytics - Sentiment & keywords
• 🔍 Search - Query scraped data
• 💬 Comments - Comment analysis
• ⚙️ Scraper - Start new scrapes
• 📋 Job History - View all jobs
• 🔌 Integrations - API, export, plugins

🚀 REST API

python main.py --api
# API at http://localhost:8000
# Docs at http://localhost:8000/docs

Endpoints:

Endpoint	Description
GET /posts	List posts with filters
GET /comments	List comments
GET /subreddits	All scraped subreddits
GET /jobs	Job history
GET /query?sql=...	Raw SQL queries
GET /grafana/query	Grafana time-series

📦 Export & Maintenance

# Export to Parquet (for DuckDB/warehouses)
python main.py --export-parquet python

# View job history
python main.py --job-history

# Backup database
python main.py --backup

# Optimize database
python main.py --vacuum

📅 Scheduled Scraping

# Scrape every 60 minutes
python main.py --schedule delhi --every 60

# With options
python main.py --schedule delhi --every 30 --mode full --limit 50

🔍 Search & Analytics

# Search scraped data
python main.py --search "credit card" --min-score 100

# Run sentiment analysis
python main.py --analyze delhi --sentiment

# Extract keywords
python main.py --analyze delhi --keywords

---

🐳 Docker

Quick Start

# Build
docker build -t reddit-scraper .

# Run scrape
docker run --env-file .env -v ./data:/app/data reddit-scraper python --limit 100

# Run with plugins
docker run --env-file .env -v ./data:/app/data reddit-scraper python --plugins

Docker Compose (Full Stack)

# Start API + Dashboard
docker-compose up -d

# Access:
# Dashboard: http://localhost:8501
# API: http://localhost:8000/docs

Deploy to AWS/VPS

# SSH into your server
ssh user@your-server-ip

# Clone repo
git clone https://github.com/ksanjeev284/reddit-universal-scraper.git
cd reddit-universal-scraper

# Start services
docker-compose up -d

# Open firewall ports
sudo ufw allow 8000
sudo ufw allow 8501

Access:

• http://your-server-ip:8501 → Dashboard
• http://your-server-ip:8000/docs → API

---

🔗 External Integrations

Metabase

1. Start API: python main.py --api
2. Add HTTP datasource: http://localhost:8000
3. Query: /posts?subreddit=python&limit=100

Grafana

1. Install "JSON API" or "Infinity" plugin
2. Add datasource: http://localhost:8000
3. Use /grafana/query for time-series

DuckDB

import duckdb

# Export to Parquet first
# python main.py --export-parquet python

# Query directly
duckdb.query("SELECT * FROM 'data/parquet/*.parquet'").df()

---

📁 Project Structure

reddit-scraper/
├── main.py              # CLI entry point
├── config.py            # Settings
├── analytics/           # Sentiment & keywords
├── alerts/              # Discord/Telegram
├── api/                 # REST API server
├── dashboard/           # Streamlit UI
├── export/              # Database & exports
├── plugins/             # Post-processing plugins
├── scheduler/           # Cron scheduling
├── search/              # Search engine
└── data/
    ├── r_subreddit/     # Scraped data
    ├── backups/         # DB backups
    └── parquet/         # Parquet exports

---

📊 Data Output

posts.csv

Column	Description
id	Reddit post ID
title	Post title
author	Username
score	Net upvotes
num_comments	Comment count
post_type	text/image/video/gallery
selftext	Post body
sentiment_score	-1.0 to 1.0 (with plugins)

comments.csv

Column	Description
comment_id	Comment ID
post_permalink	Parent post
author	Username
body	Comment text
score	Upvotes

---

⚙️ Environment Variables

# Notifications
export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
export TELEGRAM_BOT_TOKEN="123456:ABC..."
export TELEGRAM_CHAT_ID="987654321"

---

📜 License

MIT License - Feel free to use, modify, and distribute.

🤝 Contributing

Pull requests welcome! For major changes, please open an issue first.