Open source

We build web scraping tools in the open

Zyte started as the team behind Scrapy, and open source is still how we work. Zyte engineers maintain Scrapy and the libraries underneath it, fixes land upstream before they reach our own products, and what we publish stays free to use, commercially or not.

Browse librariesJoin the community

Paid maintainers

Core Scrapy maintainers are Zyte engineers, on the clock.

Upstream first

What we build for customers lands in the public repos.

Free, always

BSD-licensed, no paywalled features, no rug pulls.

Flagship projects

The tools we maintain

The projects we lead, use in production every day, and support publicly.

The leading web scraping framework

Scrapy

The open source framework for web crawling at scale, created by Zyte's co-founders and maintained by our engineers. An async engine, batteries-included middlewares, and a plugin ecosystem the whole industry builds on.

  • Async crawling engine, built for scale
  • Middlewares, pipelines and extensions
  • Created and maintained by Zyte engineers
64,394pablohoffmanpablohoffmanwRARwRARdangradangrakmikekmikeGallaecioGallaecio+369$ pip install scrapy

  • Parsing architecture

    web-poet & scrapy-poet

    Page objects for web scraping: keep extraction logic separate from the crawler so it can be reused, unit tested, and swapped per site without touching your spiders.

    • Extraction logic decoupled from crawlers
    • Unit-testable parsers
    • Swap providers per site
    106kmikekmikewRARwRARBurnzZBurnzZGallaecioGallaeciovictor-torresvictor-torres+8

  • Monitoring & QA

    Spidermon

    Battle-tested monitoring for Scrapy. Validate the items you extract, assert on job-level stats, and get alerted the moment a spider quietly starts returning junk.

    • Schema validation on extracted items
    • Job stat monitors and thresholds
    • Slack, email and custom alerts
    562gatufogatuforennerocharennerochajesuslosadajesuslosadafurther-readingfurther-readingVMRuizVMRuiz+54

  • Zyte API integration

    scrapy-zyte-api

    The official Scrapy plugin for Zyte API. Unblocking, headless browser requests and AI extraction from inside your existing spiders, with sessions and retries handled for you.

    • Unblocking and sessions handled
    • Headless browser requests
    • AI extraction from your spiders
    43GallaecioGallaeciowRARwRARBurnzZBurnzZkmikekmikesortafreelsortafreel+12

  • Browser automation

    scrapy-playwright

    Playwright, wired into Scrapy's download layer. Render JavaScript-heavy pages, click and scroll through them, and keep the rest of your project unchanged.

    • Render JavaScript-heavy pages
    • Scripted clicks, scrolls and waits
    • Drop-in download handler
    1,447elacuestaelacuestawRARwRARGallaecioGallaeciohmtbrhmtbraxiom-of-choiceaxiom-of-choice+9

  • Selectors · Python

    Parsel

    The selector library at the heart of Scrapy, usable on its own. Query HTML and XML with XPath or CSS, chain the two, and pull out text or attributes with a couple of lines.

    • XPath and CSS selectors, chainable
    • Works standalone, no Scrapy needed
    • The engine behind every Scrapy spider
    1,354wRARwRARGallaecioGallaecioeliasdorneleseliasdorneleskmikekmikepablohoffmanpablohoffman+50

  • Open standard · AI agents

    Agent Skills

    Portable, openly packaged skills that teach any coding agent how Zyte engineers build scrapers: modern Scrapy patterns, page objects and pytest fixtures instead of hallucinated selectors.

    • Works with any Agent Skills-compatible agent
    • Scrapy + web-poet scaffolding, not guesswork
    • Free to install and use
    9github-actions[bot]github-actions[bot]

Also ours

Smaller libraries, doing the unglamorous work

Pieces of Scrapy we release on their own, so you can use them anywhere.