Agentic Web Data

Built for real-world scraping

Bring web data extraction into your IDE or coding assistant.
Access, extract, and structure data as part of your normal workflow.

Try our Claude Code Plugin

Why a plugin?

Claude tries to write a Scrapy spider. With the Zyte skill, it gets it right.

Bare Claude knows Python. It doesn't know Scrapy the way Zyte engineers do. The plugin teaches it — async patterns, scrapy-poet page objects, the right libraries, real tests, anti-bot wired in. The spider that comes out of the prompt actually runs. Describe data. Get data.

Claude on its own

Generic AI coding assistant

With Zyte skills

Domain-expert AI system

A spider that runs first time

Get a runnable Scrapy spider for a new site without correction turns.

Gets modern Scrapy patterns wrong

Writes simple & non testable code, and fails to include key addons.

Spider runs first time.

Correct async Scrapy patterns, scrapy-poet page objects, targeted selectors, testable code with pytest.

Code that holds up to review

Extraction shaped like a senior scraping engineer would write it.

Best-guess extraction

Often decides your data shape for you, and hallucinates selectors, leaving you with questionable quality.

Built to Zyte's quality bar

Code shaped like the spiders Zyte engineers ship — accurate fields, clean patterns, scalable production spiders.

Anti-bot handled out of the box

No 403s and no unblocking layer to write yourself.

No anti-bot solution

You wire one in by hand or watch the spider fail at the first ban.

Zyte API wired in by default

Bring your own if you prefer, but it works out of the box.

Testable code generation

pytest to guarantee your code.

Doesn't know how to create HTML fixtures.

No tests, or tests built.

Tests scaffolded with the spider

pytest tests scaffolded against saved HTML fixtures. Tests fail? Prompt a fix and run again.

Scrapy stack awareness

The right library at the right version, wired through correctly with our most important addons.

Produces generic Scrapy.

Doesn't know scrapy-poet, zyte-common-items, scrapy-zyte-api, or the addons system.

Reaches for the right library and version

Wires them through the addons system, not legacy middleware.

Agentic Web Data

Adding 15 years of web scraping expertise to your AI workflow

Claude Code Plugin

Turn prompts into production-ready web data pipelines. Skills to build a complete Scrapy project from just a URL and data type.

Install →Read the docs

Web Scraping Copilot

VS Code extension that generates extraction logic and helps with project creation in your favorite IDE.

Install →Learn more

Coming soon

Cursor Plugin

Coming soon

Codex Plugin

Coming soon

VS Code Plugin

Tutorials

From installation to working spider in minutes

Claude Code Plugin

See how the Zyte Claude Code plugin turns a website URL into a working, tested Scrapy scraper in under 15 minutes — ready to deploy and run.

Web Scraping Copilot

Web Scraping Copilot is built to reduce friction between idea and extraction. No context switching. No black boxes. No hidden automation.

Explore the documentation

Get started with Zyte plugins, from setup to building production-ready scraping workflows.

Read the docs

Insights from web data experts

Explore opinions, lessons, and emerging approaches to extracting and managing web data at scale.

Introducing Web Scraping Copilot 1.0

Discover Web Scraping Copilot 1.0, Zyte’s VS Code extension that uses AI to generate, test, and deploy production-ready Scrapy spiders faster while maintaining full developer control.

The Scrapy whisperer: Adrian Chaves on Web Scraping Copilot

An interview with one of the popular framework’s co-maintainers on Zyte’s new, AI-powered Scrapy sidekick.

Claude skills, MCP or Web Scraping Copilot: Which should you choose?

Three ways to bring Zyte-powered web data into your AI workflow — from production spiders to conversational extraction.

FAQ - Everything you need to know

What is the scraping-agent-skills plugin?

A Claude Code plugin that converts a plain-English prompt into a fully working Scrapy spider, complete with web-poet page objects. Schema discovery, code generation, and validation all happen directly inside your AI agent session — no manual scaffolding required.

Who is it for?

Built for developers and data teams who need reliable, production-ready web extraction fast. It's especially useful if you want Scrapy + web-poet scaffolding without writing boilerplate from scratch, a reusable extraction spec before committing to any specific implementation, or a "data-first" workflow where you validate what you're extracting before generating code.

What do I need to get started?

Three things: Claude Code (CLI or desktop app), uv Python package manager (the plugin will prompt you to install it if it's missing), and a Zyte API key — recommended for fetching and rendering pages, but not strictly required to run the plugin.

How does the workflow actually work?

The process has two clearly separated stages. First, the extraction spec: the agent fetches pages, discovers fields, and iterates with you until you approve the schema and expected values. Second, code generation: the approved spec is used to produce a complete Scrapy project (page objects, test fixtures, spider).

This stage does not crawl the web — it works entirely from the outputs of stage 1. Crawling only happens when you run the generated spider locally, or deploy and run it on Scrapy Cloud.

What does the output look like?

You get two artifacts: a .scrape/{site-name}/ folder containing the extraction spec (schema, saved pages, and approved expected values), and a fully generated Scrapy project with web-poet page objects, pytest fixtures, and a wired-up spider ready to run — including an option to deploy to Scrapy Cloud.

Does it work with AI agents other than Claude Code?

Not yet — Claude Code is the only supported agent today. Support for additional agents is coming soon.

Can I use the extraction spec without the generated Scrapy code?

Yes. The spec is a technology-neutral artifact that isn't tied to any framework. It can feed a plain-Python implementation, direct Zyte API extraction, or a fully manual process. The generated Scrapy project is one possible consumer — not the only one.

How does it handle websites that change over time?

The spec is the durable part of your project. When a site changes, you re-run the spec stage (or update field selectors in the page objects) without discarding the rest of your project structure. The deliberate separation of spec from generated code is what makes the system resilient to change.

Does it work with JavaScript-heavy sites?

Yes. The plugin delegates page fetching and rendering to the Zyte API, so JavaScript-rendered content is handled transparently — no extra configuration needed.

Is a Zyte API key required?

No, the plugin can run without one. However, we recommend using the Zyte API for fetching and extraction — it handles JavaScript rendering, anti-bot measures, and reliability, giving you significantly better results on complex sites.

Is the plugin free to use?

Yes, the plugin is Free to use. However, we recommend using the Zyte API for fetching and extraction — it handles JavaScript rendering, anti-bot measures, and reliability, giving you significantly better results on complex sites.