Field notes from the world of data extraction.

Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.

The State of Web Access report cover

The State of Web Access

View the free State of Web Access report to learn just how difficult modern web scraping has become.

SoWA fsahion icon
Access handling

Fashion websites are the hardest to size up

While publishers fight over AI crawlers, fashion has built some of the most heavily defended shop windows on the web. So, which controls are in fashion, in fashion?

Robert Andrews
Dom and Neha talking about synthetic web and extract summit

The web is synthetic. Now look at the human trail

Domagoj Marić explores how AI, web scraping, and OSINT turn scattered personal data into profiles, scams, and security risks at Extract Summit.

Neha Setia Nagpal
State of Web Access - industries
Access handling

The State of Web Access: How different industries behave to bots

Zyte's large-scale audit of web access controls shows how different kinds of businesses exhibit different policies.

Robert Andrews
Fran Muñoz and Neha Setia Nagpal beside the words “From $5,000 to $70: Production-grade AI coding.”

A $70-a-month production app replaced a $5,000-a-month platform: What production-grade AI coding looks like

See how Fran Muñoz used production-grade AI coding, specification-driven development, and 3,600 tests to replace a $5,000-a-month platform with a $70 app.

Neha Setia Nagpal
podcast-10

Podcast Episode 10 is out: Skills, packages & updates

Episode 10 of the Zyte podcast is out, and it is one of the widest-ranging conversations we have recorded this year. John Rooney sits down with Neha Setia Nagpal and Ayan Pahwa for an unfiltered, all-in-style chat covering everything the team has been building, reading, and arguing about lately.

John Rooney
fingerprint-changes

Chrome has a new potential fingerprint vector

navigator.cpuPerformance is coming to Chrome in August, v152. What does it mean for fingerprinting and will it mean changes to scraping stacks?

John Rooney
SoWA: Websites locked
Access handling

The State of Web Access: How today's web raises its barriers

The largest ever audit of web access control mechanisms lifts the lid on how sites are regulating programmatic visits.

Robert Andrews
zyte-agent-tool

Harness Engineering, part 4: giving your agent a custom fetch tool that survives the real web

Harness engineering part 4: build a custom fetch tool with the Claude Agent SDK and Zyte API so your agent gets structured web data without getting blocked.

Ayan Pahwa
spidey-sense-1

Meet scrapy-spidey-sense: A preflight check for Scrapy spiders

`scrapy-spidey-sense` is an open-source command-line interface (CLI) that checks a Scrapy project before the crawl begins. It performs local, static analysis, reports the production-readiness basics that are present or missing, assigns a score, and connects each finding to a practical fix or relevant documentation

Neha Setia Nagpal
scrapy-series-4
Scraping strategy

Rendering JavaScript pages without giving up Scrapy

How to scrape JavaScript pages with Scrapy, checking the response and API first, then adding scrapy-playwright only for the requests that need a browser.

John Rooney
spidermon-part-2
Web data collection

Scrapy spider monitoring made easy with Spidermon

How do you know you're collecting all the data you need? And how can you be sure it's actually what you were expecting? Use Spidermon.

John Rooney