Field notes from the world of data extraction.

Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.

scrapy-ai-skills
Scraping practice

AI generated these Scrapy projects - why I won't ship them

What happens if you let AI create a Scrapy project from just a simple prompt? Here's what I got and what I had to fix.

John Rooney1 min read
Harness Engineering, part 2: harnessing a data extraction agent
Large Language Models (LLMs)

Harness Engineering, part 2: harnessing a data extraction agent

Point an agent at a website, name the fields, get structured records back. Harness engineering part 2: the harness decisions that make a data extraction agent reliable.

Ayan Pahwa8 min read
Harness Engineering, part 1: what is an agent harness and why it matters
Large Language Models (LLMs)

Harness Engineering, part 1: what is an agent harness and why it matters

Same model, same weights, zero retraining — LangChain changed nothing but the scaffolding around a coding model and jumped it from 30th place to the top five on a benchmark. That scaffolding has a name: the harness. And it's the part you actually control.

Ayan Pahwa8 min read
The harness matters more than the model - Podcast EP07
AI-assisted data extraction

The harness matters more than the model - Podcast EP07

"The model is the engine — but the harness is everything else." In Episode 7, we dig into why the infrastructure layer around your AI model matters more than the model itself, rank the best models available right now, and ask whether the open-weighted revolution is about to make frontier subscriptions obsolete.

John Rooney11 min read
Zyte's first Developer Community Meetup: the recap, slides, and recording
Announcement

Zyte's first Developer Community Meetup: the recap, slides, and recording

AI agents can now write, run, and self-heal your web scrapers, and in Zyte's first-ever Web Scraping Community Meetup we show you exactly how. Live demos, a Claude Code plugin that turns a prompt into production-ready data, and a fireside chat on where AI is really heading.

Ayan Pahwa3 min read
1 C N T 1225

How to run any model inside Claude Code

I run GLM 5.2 inside Claude Code with the same tools, the same skills, and the same agent loop. Three environment variables is all it takes.

Ayan Pahwa17 min read
C N T 1216 Main
AI

The best agent skill is the one that says the least

More instruction, worse output. Zyte's head of R&D on why telling your agent exactly what to do can blind it to the obvious answer.

Neha Setia Nagpal14 min read
Why I'm adding GLM-5.2 to my agentic coding arsenal
Large Language Models (LLMs)

Why I'm adding GLM-5.2 to my agentic coding arsenal

Ayan Pahwa ran GLM-5.2 against Claude Sonnet 4.6 and Opus 4.8 on real scraping work: parity on extraction, 8.5x cheaper on planning, slower and wordier.

Ayan Pahwa14 min read
How to build your first Scrapy extension
Developer interest

How to build your first Scrapy extension

My Scrapy project now plays a triumphant fanfare when a crawl finishes clean and a sad trombone when it doesn't, which is also how I finally learned what Scrapy's extension points are actually for.

Ayan Pahwa11 min read
My personal agent setup: the architecture that runs my household and my DevRel work
AI

My personal agent setup: the architecture that runs my household and my DevRel work

"Four people, four diets, two work schedules, and a baby who answers to nobody. That's what finally made me build a personal agent." A walkthrough of the actual architecture I run to hold my household and my DevRel work together — profiles, skills, memory, and the web-data layer that makes it all reach the live web.

Neha Setia Nagpal11 min read
Data is the new water
Web data collection

Data is the new water

For 20 years, we were told “data is the new oil”. That’s no longer true. In the era of the fluid web, information is more fundamental, cleaner and abundant than that.

Robert Andrews10 min read
Building robust agentic AI workflows with rapid web data
Use case

Building robust agentic AI workflows with rapid web data

AI agents need access to public web data, right now. Tools connected to web scraping APIs empower agents to return live data quickly.

Theresia Tanzil10 min read