Python 的高层网页抓取框架，帮你快速写爬虫，自动处理请求调度、页面解析和数据提取，适合大规模采集任务。
把任意网页抓取并转成干净 Markdown 或结构化数据，直接喂给 LLM 和 AI Agent，省去自己写爬虫和清洗的麻烦。
用大语言模型驱动的 Python 爬虫，只需描述想抓什么，就能自动完成网页抓取和结构化数据提取，省去手写解析规则。
…ntrolling a running crawl over HTTP, used by the [Scrapy MCP server](https://docs.scrapy.org/en/latest/topics/agents.html using-the-mcp-server) - Experimental aiohttp -ba
…llowed TLS versions [Full Changelog](https://docs.scrapy.org/en/latest/news.html scrapy-2-17-0-2026-07-07)
…for Twisted 26.4.0+ [Full Changelog](https://docs.scrapy.org/en/latest/news.html scrapy-2-16-0-2026-05-19)
…ks by @pawelmhm in https://github.com/scrapinghub/scrapyrt/pull/158 Make the errback handling method configurable by @radostyle in https://github.com/scrapinghub/scrapyrt
fix compatibility with scrapy 2.10 https://github.com/scrapinghub/scrapyrt/pull/150 drop end of life pythons (3.6, 3.7) add python 3.11 support
add support for specifying Twisted reactor in Scrapy settings or ScrapyRT settings, fixes https://github.com/scrapinghub/scrapyrt/issues/132 drop Python 3.5 support add P