Open source
We build web scraping tools in the open
Zyte started as the team behind Scrapy, and open source is still how we work. Zyte engineers maintain Scrapy and the libraries underneath it, fixes land upstream before they reach our own products, and what we publish stays free to use, commercially or not.
Paid maintainers
Core Scrapy maintainers are Zyte engineers, on the clock.
Upstream first
What we build for customers lands in the public repos.
Free, always
BSD-licensed, no paywalled features, no rug pulls.
Flagship projects
The tools we maintain
The projects we lead, use in production every day, and support publicly.
The leading web scraping framework
Scrapy
The open source framework for web crawling at scale, created by Zyte's co-founders and maintained by our engineers. An async engine, batteries-included middlewares, and a plugin ecosystem the whole industry builds on.
- Async crawling engine, built for scale
- Middlewares, pipelines and extensions
- Created and maintained by Zyte engineers
pablohoffman
wRAR
dangra
kmike
Gallaecio+369$ pip install scrapyParsing architecture
web-poet & scrapy-poet
Page objects for web scraping: keep extraction logic separate from the crawler so it can be reused, unit tested, and swapped per site without touching your spiders.
- Extraction logic decoupled from crawlers
- Unit-testable parsers
- Swap providers per site
Monitoring & QA
Spidermon
Battle-tested monitoring for Scrapy. Validate the items you extract, assert on job-level stats, and get alerted the moment a spider quietly starts returning junk.
- Schema validation on extracted items
- Job stat monitors and thresholds
- Slack, email and custom alerts
Zyte API integration
scrapy-zyte-api
The official Scrapy plugin for Zyte API. Unblocking, headless browser requests and AI extraction from inside your existing spiders, with sessions and retries handled for you.
- Unblocking and sessions handled
- Headless browser requests
- AI extraction from your spiders
Browser automation
scrapy-playwright
Playwright, wired into Scrapy's download layer. Render JavaScript-heavy pages, click and scroll through them, and keep the rest of your project unchanged.
- Render JavaScript-heavy pages
- Scripted clicks, scrolls and waits
- Drop-in download handler
Selectors · Python
Parsel
The selector library at the heart of Scrapy, usable on its own. Query HTML and XML with XPath or CSS, chain the two, and pull out text or attributes with a couple of lines.
- XPath and CSS selectors, chainable
- Works standalone, no Scrapy needed
- The engine behind every Scrapy spider
Open standard · AI agents
Agent Skills
Portable, openly packaged skills that teach any coding agent how Zyte engineers build scrapers: modern Scrapy patterns, page objects and pytest fixtures instead of hallucinated selectors.
- Works with any Agent Skills-compatible agent
- Scrapy + web-poet scaffolding, not guesswork
- Free to install and use
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?v=4){kind=small}
Also ours
Smaller libraries, doing the unglamorous work
Pieces of Scrapy we release on their own, so you can use them anywhere.
