Scraping strategy

Articles from the Zyte blog about Scraping strategy.

Illustration of an e-commerce storefront secured inside a vault
Access handling

The world's hardest sites protect their prices

Only a sliver of the web demands the heaviest machinery to read. It is dominated by shops, property listings and booking engines, where every published price is somebody's market intelligence.

Robert Andrews
Isometric illustration of a website as a portal with an open, glowing door
Access handling

The easiest sites to scrape are the ones that want to be found

Want an easy scrape? Skip the shops and try the military. Defence, government and science sites rank among the most open on the web.

Robert Andrews
Group on stage at Extract Summit Dublin 2026
Web data collection

Five takeaways from Extract Summit Dublin

AI is making scraping code cheaper to write. Extract Summit Dublin explored the harder work that remains: access, infrastructure, maintenance and trust.

Robert Andrews
Inside Scrapy's RemoteControl extension
How To

Inside Scrapy's RemoteControl extension

Scrapy 2.19's RemoteControl extension runs a small authenticated HTTP server inside every crawl. How it starts, how it's secured, what the code you send can reach, and what you can do with it on a live crawl, plus the traps and what changes under scrapy-zyte-api.

John Rooney
Theresia Tanzil, State of Web Access portrait
Scraping strategy

The web is being priced, not blocked

Zyte's State of Web Access research finds new barriers making web scraping more difficult. We interview the researcher who says that doesn't mean the opportunity is over.

Robert Andrews
scrapy-series-4
Scraping strategy

Rendering JavaScript pages without giving up Scrapy

How to scrape JavaScript pages with Scrapy, checking the response and API first, then adding scrapy-playwright only for the requests that need a browser.

John Rooney
scrapy-items-types
Scraping strategy

A guide to Scrapy item types

Scrapy supports several item types, from plain dicts to Item objects, dataclasses and attrs classes. What each one gives you and which to use in your spiders.

Ayan Pahwa
podcast-ep08
Scraping strategy

Podcast Ep08 - Scrapy, Python and mushroom soup

Episode 8 of the Zyte Scrapy podcast, on why Scrapy's core leaves browsers, data shaping and anti-bot handling out, and how you add them when you need them.

John Rooney
skills-are-software
Open-source

Treat your AI skills like software, starting with evals

Most AI skills are never tested — and it shows. Here's how Zyte evaluates scraping skills like real software, catching failures demos miss.

Neha Setia Nagpal14 min read
The great wall of data: The complexities of web scraping in the Asian market
Scraping strategy

The great wall of data: The complexities of web scraping in the Asian market

While the technological arms race of web data access is universal, the battleground in Asia has its own unique rules of engagement.

Theresia Tanzil10 min read
Brand visibility in the digital era: How web data help brands see the full picture
Web data collection

Brand visibility in the digital era: How web data help brands see the full picture

Discover how web data helps brands improve visibility, track competitors, monitor availability, and analyze reviews to win on the digital shelf.

Theresia Tanzil5 min read
How online retailers use web data to compete on price, promotion, and availability
Web data collection

How online retailers use web data to compete on price, promotion, and availability

Discover how retailers leverage web data to optimize pricing, track competitor stock, detect trends, and improve sales performance.

Theresia Tanzil5 min read

More articles on Scraping strategy