Field notes from the world of data extraction.

Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.

ayan-pahwa-real-python-podcast

I joined the Real Python podcast to talk harnesses and Scrapy

On the Real Python podcast with Christopher Bailey: which matters more for an AI agent, the model or the harness around it, and where Scrapy fits in.

Ayan Pahwa
scrapy-series-2

Building maintainable spiders with scrapy-poet

Separating your extract and parsing logic out help increase the maintainability and extensibility of your projects, and scrapy-poet makes it easy.

John Rooney
scrapy-series-1

Modern Scrapy for experienced developers: A new series

How to create production ready Scrapy projects to scrape the modern web. In this article we start the process of creating our spider, look at settings, and build a pipeline to help keep our data quality high.

John Rooney
scrapy-items-types
Scraping strategy

A guide to Scrapy item types

Scrapy supports several item types, from plain dicts to Item objects, dataclasses and attrs classes. What each one gives you and which to use in your spiders.

Ayan Pahwa
Agentic Web Data toolbelt
AI-assisted data extraction

New Zyte add-ons: Agent Skills, Codex, GitHub and more updates

A host of additions to Zyte’s agentic scraping toolbelt helps developers go from prompt to working spider code and beyond.

Valter Sciarrillo10 min read
podcast-ep08
Scraping strategy

Podcast Ep08 - Scrapy, Python and mushroom soup

Episode 8 of the Zyte Scrapy podcast, on why Scrapy's core leaves browsers, data shaping and anti-bot handling out, and how you add them when you need them.

John Rooney
Extract Summit - Zyte staff

Five reasons to attend this year’s Extract Summit

In its eighth year, Extract Summit is again shaping up to be the web data industry's premier gathering.

Robert Andrews
scrapy-playwright
Scraping practice

The missing middle ground in scrapy-playwright just got filled

You can now choose which Python browser library you want to use with scrapy-playwright. I go through why this is a huge deal for the right scraping demographic.

John Rooney8 min read
harness-engineering-3
AI

Harness Engineering #3- Headless mode: the minimal agent harness

Harness engineering part 3: headless mode runs a coding agent without the chat. Point it at a web page and get structured data back in about ten lines.

Ayan Pahwa12 min read
An automatic Claude writer device, writing code.
AI-assisted data extraction

Claude Fable 5 is the new best model for writing scrapers

We ran nine models, including the new GPT-5.6, through our Zyte Scraping Code Benchmark. Claude Fable 5 puts Sol and others in the shade - but it’s pricey, and the best extraction code still depends on the best infrastructure.

Konstantin Lopukhin8 min read
gpt5-6-cover
AI

GPT-5.6, Fable 5, and GLM-5.2 entered a bar “crawl” and got hit by The Rate Limit

When OpenAI shipped GPT-5.6 and Anthropic's Fable 5 was sitting at the top of the price list, my question was not "which one wins a leaderboard." It was the one I actually pay for: for the scraping I do, how much model do I need to buy?

Ayan Pahwa16 min read
skills-are-software
Open-source

Treat your AI skills like software, starting with evals

Most AI skills are never tested — and it shows. Here's how Zyte evaluates scraping skills like real software, catching failures demos miss.

Neha Setia Nagpal14 min read