Field notes from the world of data extraction.

Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.

Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?
Developer interest

Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?

A viral clip from Claude Code's creator put a name to something a lot of us have been circling: loop engineering. Here's why web scraping may be its best-fit domain — and what that means in practice.

Ayan Pahwa12 min read
Enhancing AI model performance with fresh web data
Use case

Enhancing AI model performance with fresh web data

No-one likes an out-of-touch AI assistant. Fortunately, rapid refreshing can keep AI models aware of the very latest public information.

Theresia Tanzil10 min read
What's becoming of web scraping developers in the age of AI agents?
AI-assisted data extraction

What's becoming of web scraping developers in the age of AI agents?

AI agents can generate code, suggest selectors, and draft crawl logic. What they can't do is design the system that decides when to stop, what to trust, and how to recover when the web pushes back. That job still belongs to a human.

Neha Setia Nagpal14 min read
Building superior AI models with quality web data

Building superior AI models with quality web data

Training data quality can make or break AI model effectiveness. So how are engineers sourcing the web’s best input?

Theresia Tanzil10 min read
Web scraping on an iPhone? Yes, really!
Use case

Web scraping on an iPhone? Yes, really!

When you can scrape the web by API, a world of possibility opens up. Yes, you can extract live web data using iOS Shortcuts.

Ayan Pahwa10 min read
Introducing Zyte Web Data for Claude Code: Production-ready scraping from a prompt
Use case

Introducing Zyte Web Data for Claude Code: Production-ready scraping from a prompt

Developers are embracing agentic coding tools - but data engineers need tools with specialist scraping skills.

Valter Sciarrillo10 min read
Copilot like a pro: Eight tips that supercharged my workflow
Use case

Copilot like a pro: Eight tips that supercharged my workflow

AI-assisted coding is a revelation. But are you getting the most out of your IDE’s sidebar sidekick?

Shivanshu Srivastava10 min read
Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init
Developer interest

Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init

Your VPS is ready, but now you need to work through the same sequence you have run a dozen times before: apt update, apt install python3-pip, pip install scrapy, playwright install chromium, the Chromium dependency list that never installs cleanly on the first try, Redis, possibly Postgres, whatever else this particular project needs.

Ayan Pahwa9 min read
What multi-agent orchestration looks like in a large-scale web scraping project
AI-assisted data extraction

What multi-agent orchestration looks like in a large-scale web scraping project

Multi-agent orchestration is having its moment. The diagrams are everywhere now. Boxes for planners, boxes for hands, boxes for daemons, arrows to a shared brain, a human floating at the top. They keep getting prettier. The part where the web pushes back is still the part nobody draws.

Neha Setia Nagpal18 min read
What are residential proxies banner
Use case

What is a residential proxy?

Learn what residential proxies are, how they compare to datacenter proxies, and why modern web scraping needs more than IP diversity.

Arnold Alexander10 min read
Zyte Blog — field notes from the world of data extraction

How much do rotating proxies cost?

Learn how much rotating proxies cost, what affects pricing, and why total web scraping costs often go beyond proxy subscriptions.

10 min read
Zyte Blog — field notes from the world of data extraction
Use case

How do rotating proxies work?

Learn how rotating proxies work, when to use them for web scraping, and why IP rotation alone is not enough for reliable data access.

Arnold Alexander10 min read