Field notes from the world of data extraction.
Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.
The Scrapy whisperer: Adrian Chaves on Web Scraping Copilot
An interview with Scrapy maintainer Adrian Chaves on Zyte’s Web Scraping Copilot, AI-generated parsing code, and building reliable scraping workflows.
How to parse HTML tables into structured data (CSV/Excel)
In this guide, you'll learn three things: how HTML tables are actually structured (so the parsing makes sense), how to extract clean tabular data using Python, and how to export it to CSV or Excel
Code is cheap, show me the talk: How copilots are re-engineering developers
Discover how AI copilots like Zyte’s Web Scraping Copilot are transforming developer workflows—making code a commodity and shifting value to problem-solving and prompting skills.
Web scraping finally has a home in the IDE
Discover how web scraping is moving into the IDE. Learn how tools like VS Code and AI-assisted extensions are streamlining scraper development, testing, and maintenance.
Introducing Web Scraping Copilot 1.0: AI-accelerated web scraping inside VS Code
Discover Web Scraping Copilot 1.0, Zyte’s VS Code extension that uses AI to generate, test, and deploy production-ready Scrapy spiders faster while maintaining full developer control.
How to Test Web Scrapers During Development
Learn how to test web scrapers during development. Validate selectors, use HTML fixtures, and ensure reliable data extraction across changing websites.
How Developers Debug Web Scraping Selectors
Learn how developers debug web scraping selectors. Discover common issues, testing techniques, and how to build reliable extraction logic for changing websites.
Best VS Code Extensions for Web Scraping
Discover the best VS Code extensions for web scraping, including Python tools, HTTP clients, and AI-powered solutions to build and debug scrapers faster.
How to Build a Web Scraper in VS Code (Step-by-Step)
Learn how to build a web scraper in VS Code using Scrapy and AI tools. Follow this step-by-step guide to create, test, and scale your scraping projects.
Build your own MCP server: LLMs meets web data with Zyte API
Learn how to build your own Model Context Protocol (MCP) server to connect LLMs with real-time web data using Zyte API, FastMCP, and the Docker MCP toolkit.
More data, more trouble: How a perfect corpus corrupted my AI dream
A failed AI experiment reveals why adding more data doesn’t always improve LLM outputs. Learn when web scraping, RAG, and curated datasets actually make AI better.
Stop using Python requests for web scraping: Use these modern modules instead
While the 'Requests' library remains the default choice for many Python developers due to its reliability and extensive documentation, the Python HTTP landscape has evolved considerably. Modern alternatives now offer significant advantages, including built-in asynchronous support, HTTP/2 compatibility, enhanced performance, and up-to-date TLS handling.
_HFpro5d6k3.png&w=256&q=75)
_E4PyVpfAxa.png&w=256&q=75)
-(1).png&w=1920&q=75)
-(1)_VZGHqxCgXV.png&w=1920&q=75)
