Web data collection

Articles from the Zyte blog about Web data collection.

Isometric illustration of a website as a portal with an open, glowing door
Access handling

The easiest sites to scrape are the ones that want to be found

Want an easy scrape? Skip the shops and try the military. Defense, government and science sites rank among the most open on the web.

Robert Andrews
Group on stage at Extract Summit Dublin 2026
Web data collection

Five takeaways from Extract Summit Dublin

AI is making scraping code cheaper to write. Extract Summit Dublin explored the harder work that remains: access, infrastructure, maintenance and trust.

Robert Andrews
A line-up of robot babushka or Matryoshka dolls.
Access handling

What web access actually costs - the tier picture

The barrier headlines sound extreme, but the cost data is more differentiated: 87.9% of top sites are Simple or Easy to access - and that means cost-effective.

Robert Andrews
A shopper faces an overwhelming cascade of ecommerce T-shirt product cards in many colours and configurations, illustrating product-variant explosion.
Web data collection

The trouble with variants: Solving the web's infinite ecommerce explosion

Product variants can turn a straightforward ecommerce scraping project into millions of combinations. Learn how to classify, scope and control them.

Robert Andrews12 min read
Two runners carrying backpacks full of web page cards race along a purple track towards a glowing digital finish line.
Web data collection

Need for speed: How data inertia kills business ambition

Slow web data pipelines can hold back new markets, conversion gains and product launches.

James Kehoe8 min read
A stack of six in fortifies a wall blocking a computer from accessing pages.
Access handling

The web's layered defences - how access barriers combine

Individual barriers may get the headlines, but it's stacked access control mechanisms that pose the biggest block.

Robert Andrews
Akshay Philar speaking at Extract Summit about the anti-ban arms race and the open web.
Anti-ban

The anti-ban arms race: Agentic traffic and the fight for the open web

As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.

Akshay Philar9 min read
A key made of a fingerprint is examined by a cryptographic light as it gains access to the network.
Access handling

TLS fingerprinting, the handshake that reveals your true intent

TLS fingerprinting fires before you have even asked for a page, and tells you nothing when it does. The least common barrier, and the hardest to diagnose.

Robert Andrews
Missing JavaScript has made almost half a web page invisible.
Access handling

JavaScript renders almost half of sites invisible

JavaScript renders 40.6% of top sites unreadable to a plain HTTP request, not by design but as a side effect of how the modern web is built.

Robert Andrews
A speed limit sign's speed guidance is replaced with a 403 code.
Access handling

Rate limiting is a roadblock, not a speed limit

Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle.

Robert Andrews
Zyte MCP chat box connected to fetch, search and extract tools.
Data gathering for AI

Introducing Zyte MCP: Powerful web data gathering for your agent

Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.

Valter Sciarrillo7 min read
A classic toy robot waves a protest banner showing "I'm not a robot"
Access handling

CAPTCHA is more than just a puzzle piece

The famously awkward security mechanism has quietly become an invisible behavioral scoring system. Two vendors enable 89% of CAPTCHAs, and the smallest sites use it most.

Robert Andrews

More articles on Web data collection