Field notes from the world of data extraction.

Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.

Zyte Blog — field notes from the world of data extraction

How to Scrape Search Engine Results

From SEO audits to market intelligence, scraping search engine results data can give you the insights you need to make smarter, faster business decisions.

5 min read
Why the Best Engineers Are Actually Lazy
Web scraping APIs

Why the Best Engineers Are Actually Lazy

There’s a popular archetype in web scraping circles: the heroic engineer who fights CAPTCHAs at 3 a.m., hand-tunes a proxy farm before breakfast, then rewrites four spiders after lunch because the target sites pushed new JavaScript.

Robert Andrews2 min read
7 Myths About Web Scraping APIs – Busted
Web data collection

7 Myths About Web Scraping APIs – Busted

Despite their benefits, web scraping APIs are sometimes misunderstood. So, let’s debunk some of the most common myths.

Daniel Cave7 min read
When DaaS met SaaS - the new hybrid data economy
Web data collection

When DaaS met SaaS - the new hybrid data economy

The emergence of hybrid DaaS/SaaS models is a logical response to the complex realities of modern data needs.

Debbie Reeve-Crook7 min read
How price extraction is fuelling insights for modern retailers
Web data application

How price extraction is fuelling insights for modern retailers

Retail pricing has long combined data, experience, and instinct – but today’s market volatility demands a faster, smarter approach.

Theresia Tanzil7 min read
Four sweet spots for AI in web scraping
Large Language Models (LLMs)

Four sweet spots for AI in web scraping

Discover how AI and LLMs are enhancing web scraping with smarter crawling, fuzzy data extraction, automated spider generation, and intelligent QA.

Theresia Tanzil7 min read
Kill your product - why sacrificing your cash cow can be the path to growth
Web data collection

Kill your product - why sacrificing your cash cow can be the path to growth

Letting go of the past is the best way to embrace the future. Retiring a flagship product isn’t a sign of failure; it’s a commitment to innovation.

6 min read
From script to system: 10 building blocks to scale web scraping
Web data collection

From script to system: 10 building blocks to scale web scraping

Scaling your business’ web data gathering – acquiring, monitoring and storing a growing amount of data from a growing number of sources over time – requires holistic planning.

Theresia Tanzil10 min read
Zyte Blog — field notes from the world of data extraction

Scrape Web Pages and Files Using Python, wget, and Zyte

The command-line utility wget (pronounced "web-get") can download online files. This free network downloader may run in the background without user intervention.

7 min read
New in Zyte: Scroll Control, Lower Costs, and More
Web data collection

New in Zyte: Scroll Control, Lower Costs, and More

As the web continues to evolve, Zyte API is evolving right alongside it—adding powerful new features and refinements designed to make data extraction smarter, faster, and more adaptable than ever.

Daniel Cave5 min read
The future of Scrapy: Smarter, faster and ready for AI-powered scraping
Open-source

The future of Scrapy: Smarter, faster and ready for AI-powered scraping

What does the future hold for the tool some describe as “the gift that revolutionised web scraping”?

Robert Andrews6 min read
Rise of the Data Vendor: How Outsourcing is Transforming Supply and Fuelling Businesses
Web data collection

Rise of the Data Vendor: How Outsourcing is Transforming Supply and Fuelling Businesses

With the emergence of managed data extraction vendors, businesses no longer need to gather web data themselves.

Robert Andrews6 min read