登录

scrapy

scrapyBSD-3-Clause活跃维护

Python 的高层网页抓取框架，帮你快速写爬虫，自动处理请求调度、页面解析和数据提取，适合大规模采集任务。

Scrapy, a fast high-level web crawling & scraping framework for Python.

2.20.0发布于 2026-11-06 18:23 UTC65,307star +51 / 7 天12,115fork同步于 2026-11-10 11:18 UTC
网页抓取爬虫框架Python爬虫数据采集分布式爬虫
对比 GitHub 官网crawlercrawlingframeworkhacktoberfestpython

怎么上手

由 README 提炼，执行前请确认
原文 
pip install scrapy
硬件要求按 README 说明

README 未说明

下载：源码 zip·GitHub Releases（安装包）本站不镜像安装包

项目说明

由 README 自动翻译提炼
没人工点评？写下你的经验

Python 网页爬取与结构化数据提取框架，跨平台，适合快速搭建爬虫项目

Python 的高层网页抓取框架，帮你快速写爬虫，自动处理请求调度、页面解析和数据提取，适合大规模采集任务。

核心能力

  • 从网站提取结构化数据
  • 跨平台运行
  • 由 Zyte 及社区维护
  • 提供完整文档与贡献指南

容易踩的坑

需要 Python 3.11 及以上版本；README 未给出更多使用细节，需查阅官方文档

内容由模型根据 README 生成，可能不完整或不准确，以官方文档为准。看原文 README

最近发版

5 个版本2.20.0
全部 release
  1. 2.20.0最新正式版2026-11-06 · 2026-11-06 18:23 UTC
    • Web-browser-like detection of response encodings, base URLs and meta refresh redirects, powered by frostwork
    • Requests can be awaited to get their response
    • Non-asyncio Twisted reactors are deprecated

    Full changelog

  2. 2.19.02026-09-10 · 2026-09-10 11:48 UTC
    查看更新说明
    • New RemoteControl extension which allows inspecting and controlling a running crawl over HTTP, used by the Scrapy MCP server
    • Experimental aiohttp-based download handler (now the default when running without a reactor)

    Full Changelog

  3. 2.18.02026-08-20 · 2026-08-20 15:20 UTC
    查看更新说明
    • HttpxDownloadHandler now uses httpx2
    • The Twisted-based HTTP/2 download handler is no longer experimental
    • brotli and Zstandard support are now always available, and optional extras cover the rest of the optional features
    • Late Crawler attributes, such as stats, now raise RuntimeError instead of being None before the crawl starts
    • Item exporters now export fields in declaration order
    • New MetaCopyDetectionMiddleware
    • New optimization page and built-in stats reference

    Full Changelog

  4. 2.17.02026-07-07 · 2026-07-07 10:40 UTC
    查看更新说明
    • Security bug fixes
    • HTTP/2 and SOCKS proxy support for HttpxDownloadHandler
    • Improved settings for changing allowed TLS versions

    Full Changelog

  5. 2.16.02026-05-19 · 2026-05-19 12:39 UTC
    查看更新说明
    • Official support for Python 3.14
    • Support for Twisted 26.4.0+

    Full Changelog

用户点评

登录用户的真实使用记录，可按版本和硬件筛选
共 0 条

    还没有人点评。第一个说出真实体验的人最有价值。

    README 原文

    未翻译
    在 GitHub 打开

    |logo|

    .. |logo| image:: https://raw.githubusercontent.com/scrapy/scrapy/master/docs/_static/logo.svg :target: https://scrapy.org :alt: Scrapy :width: 480px

    |version| |python_version| |tests| |coverage| |conda| |deepwiki|

    .. |version| image:: https://img.shields.io/pypi/v/Scrapy.svg :target: https://pypi.org/pypi/Scrapy :alt: PyPI Version

    .. |python_version| image:: https://img.shields.io/pypi/pyversions/Scrapy.svg :target: https://pypi.org/pypi/Scrapy :alt: Supported Python Versions

    .. |tests| image:: https://img.shields.io/github/check-runs/scrapy/scrapy/master?label=tests :target: https://github.com/scrapy/scrapy/actions?query=branch%3Amaster :alt: Tests

    .. |coverage| image:: https://img.shields.io/codecov/c/github/scrapy/scrapy/master.svg :target: https://codecov.io/github/scrapy/scrapy?branch=master :alt: Coverage report

    .. |conda| image:: https://anaconda.org/conda-forge/scrapy/badges/version.svg :target: https://anaconda.org/conda-forge/scrapy :alt: Conda Version

    .. |deepwiki| image:: https://deepwiki.com/badge.svg :target: https://deepwiki.com/scrapy/scrapy :alt: Ask DeepWiki

    Scrapy_ is a web scraping framework to extract structured data from websites. It is cross-platform, and requires Python 3.11+. It is maintained by Zyte_ (formerly Scrapinghub) and many other contributors_.

    .. _many other contributors: https://github.com/scrapy/scrapy/graphs/contributors .. _Scrapy: https://scrapy.org/ .. _Zyte: https://www.zyte.com/

    Install with:

    .. code:: bash

    pip install scrapy

    And follow the documentation_ to learn how to use it.

    .. _documentation: https://docs.scrapy.org/en/latest/

    If you wish to contribute, see Contributing_.

    .. _Contributing: https://docs.scrapy.org/en/master/contributing.html

    相关项目

    同分类里 star 最高的项目
    对比全部 →
    playwright
    microsoft·TypeScript

    用一套 API 同时驱动 Chromium、Firefox 和 WebKit 的浏览器自动化与端到端测试框架，适合写跨浏览器测试和网页抓取脚本。

    浏览器自动化+4爬虫工具
    99735+2456.8kv1.65.02026-11-10 05:40 UTC
    EasySpider
    NaiboWang·JavaScript

    可视化无代码网页爬虫，通过图形界面设计和执行采集任务，让不懂编程的人也能批量抓取网页数据。

    可视化爬虫+4爬虫工具
    44776+65.4kv0.6.52026-11-07 13:35 UTC
    cheerio
    cheeriojs·TypeScript

    用 jQuery 风格的语法在服务端解析和操作 HTML/XML，写爬虫时提取页面数据比正则和原生 DOM 顺手得多。

    HTML解析+4爬虫工具
    30536+21.7kv1.2.02026-11-10 03:19 UTC
    crawlee
    apify·TypeScript

    Node.js/TypeScript 网页抓取与浏览器自动化库，内置代理轮换和反封锁，兼容 Puppeteer、Playwright、Cheerio，可稳定采集数据供 AI、LLM、RAG 使用。

    网页抓取+4爬虫工具
    26404+371.8kv3.18.22026-11-10 11:17 UTC