Field notes from the world of data extraction.

Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.

Zyte Blog — field notes from the world of data extraction

From screenshot to shopping list in 90 seconds

I built a mood board pipeline that starts with a screenshot. Claude Skills and Zyte API for search, product extraction, and image embedding at any scale.

10 min read
Automation drives power in the data arms race
Web scraping APIs

Automation drives power in the data arms race

Anti-bot systems now evolve in minutes, not weeks. Discover why automated, self-healing scraping systems are essential to survive the 2026 data arms race and how to adapt.

Robert Andrews5 min read
Zyte Blog — field notes from the world of data extraction

Should AI Companies Build Their Own Web Scraping Pipelines?

Should AI companies build their own web scraping pipelines? Learn when in-house scraping makes sense and when it becomes costly and hard to maintain at scale.

10 min read
Zyte Blog — field notes from the world of data extraction
Use case

What Is AI Data Provenance? Definition & Importance

Learn what AI data provenance is and why it matters. Understand data origin, collection methods, governance, and how provenance supports trust and compliance.

Arnold Alexander10 min read
How web data turns e-commerce listings into retail intelligence
Web data collection

How web data turns e-commerce listings into retail intelligence

Discover how web data enables digital shelf analytics vendors to track prices, availability, and product trends at scale—fueling real-time retail intelligence and competitive advantage.

Theresia Tanzil5 min read
The seven habits of highly effective data teams
Scraping strategy

The seven habits of highly effective data teams

Discover the seven habits that set high-performing data teams apart—from treating data as a product to ensuring data trust, quality, and decision impact. Learn how leading teams scale reliable data systems.

Robert Andrews5 min read
Zyte Blog — field notes from the world of data extraction
Learn Web Scraping

How to ensure data quality in your Scrapy web scraping projects using Spidermon and Claude Code

Spidermon is an open-source monitoring framework for Scrapy. You attach it to your spider, define what "success" looks like, and it automatically checks your crawl results after the spider closes, flagging anything that doesn't meet your standards.

Ayan Pahwa5 min read
Zyte Blog — field notes from the world of data extraction

Why your API responses look like gibberish: the gzip decompression trap

The script was working. Requests were going out, responses were coming back with HTTP 200. But the response body was unreadable noise, a wall of binary characters that crashed the JSON parser and reported "no data found". No error code, no timeout, no network failure; just garbage where structured data should be.

6 min read
Dawn of the autonomous data pipeline
Use case

Dawn of the autonomous data pipeline

Discover how autonomous, agent-driven data pipelines are transforming web scraping in 2026, enabling self-healing systems, API discovery, and end-to-end automation.

Theresia Tanzil5 min read
Are programming practices relevant anymore?
Web scraping APIs

Are programming practices relevant anymore?

From LLM-powered extraction to agentic pipelines, here's how AI is reshaping every stage of the web scraping workflow in 2026 -- and what it means for your stack.

Mikhail Korobov5 min read
AI is the new engine for web scraping
Web scraping APIs

AI is the new engine for web scraping

From LLM-powered extraction to agentic pipelines, here's how AI is reshaping every stage of the web scraping workflow in 2026 -- and what it means for your stack.

Theresia Tanzil, Robert Andrews5 min read
Super-powers, toll booths and the new era of data collection
Web data collection

Super-powers, toll booths and the new era of data collection

Explore how AI is transforming web scraping, the rise of advanced anti-bot systems, and what the future holds for data collection in an increasingly controlled internet.

Fabien Vauchelles5 min read