爬虫工具

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.

Scrapy, a fast high-level web crawling & scraping framework for Python.

A visual no-code/code-free web crawler/spider易采集：一个可视化浏览器自动化测试/数据采集/网页爬虫软件，可以无代码图形化的设计和执行爬虫任务。别名：ServiceWrapper面向Web应用的智能化服务封装系统。

The fast, flexible, and elegant library for parsing and manipulating HTML and XML.

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Elegant Scraper and Crawler Framework for Golang

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

A Chrome DevTools Protocol driver for web automation and scraping.

Analysis of Bot Protection systems with available countermeasures 🚿. How to defeat anti-bot system 👻 and get around browser fingerprinting scripts 🕵️‍♂️ when scraping the web?

Twitter API Scraper | Without an API key | Twitter Internal API | Free | Twitter scraper | Twitter Bot

YouTube video downloader in javascript.

A library that scrapes Linkedin for user data

Datascraper that is currently adult focused - Updated regularly

🔮 A Node.js scraper for humans.

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Movie metadata scraper

A Powerful web scraper powered by LLM | OpenAI, Gemini & Ollama

Do you want to LEARN NEW STUFF for FREE? Don't worry, with the power of web-scraping and automation, this script will find the necessary Udemy coupons & enroll you for PAID UDEMY COURSES, ABSOLUTELY FREE!

Node.js scraper to get data from Google Play

Python library and CLI for X/Twitter scraping with multi-account rotation and built-in rate-limit handling.

Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Scrape metadata from any URL using Open Graph, JSON-LD, HTML meta tags, and smart fallbacks.

:spider: The progressive PHP crawler framework! 优雅的渐进式PHP采集框架。

TikTok LIVE API for Node.js: An unofficial library to receive live stream events (comments, gifts, etc.) in realtime from TikTok LIVE.

Transform Web Content into LLM-Ready Data

Scrape job websites into a single spreadsheet with no duplicates.

Completely free, private, UI based Tech Documentation MCP server. Designed for coders and software developers in mind. Easily integrate into Cursor, Windsurf, Cline, Roo Code, Claude Desktop App

NewPipe's core library for extracting data from streaming sites

Learn how to build your own Google Jobs scraper that simultaneously scrapes Google Jobs for multiple search queries and geo-locations with Python and Oxylabs’ Google Jobs Scraper API. https://oxylabs.io/blog/how-to-scrape-google-jobs

Free email OSINT tool, 5000+ platforms, identity clustering, breach detection. No API keys required. pip install mailaccess

Download website to local directory (including all css, images, js, etc.)

Google Play Scraper for collecting public details about applications, games, movies & TV shows, books, and more.

Python binding to Modest and Lexbor engines. Fast HTML5 parser with CSS selectors for Python.

A Telegram Mass Surveillance Bot in Python

An ergonomic, privacy-aware Python HTTP Client

Unofficial API for finviz.com

Lightweight Ruby web crawler/scraper with an elegant DSL which extracts structured data from pages.

Cinemagoer is a Python package useful to retrieve and manage the data of the IMDb (to which we are not affiliated in any way) movie database about movies, people and characters

📰 Newspaper4k a fork of the beloved Newspaper3k. Extraction of articles, titles, and metadata from news websites.

Crawly, a high-level web crawling & scraping framework for Elixir.

Scalable Python web scraping scripts for +40 popular domains

Self-hosted scraping engine — bypasses any JS challenge & captcha: Cloudflare, Turnstile, reCAPTCHA, hCaptcha, GeeTest. FlareSolverr & Byparr alternative and drop-in replacement for your *arr stack.

First ever tool to view "Instagram private posts" anonymously

An ergonomic, privacy-aware Rust HTTP Client

Unofficial MyAnimeList PHP+REST API which provides functions other than the official API

一个管理电脑内资源，并提供游戏化的综合管理器

Universal Reddit Scraper - A comprehensive Reddit scraping/archival command-line tool.