Access handling

Firewalls run the web, just not on purpose

Web Application Firewalls run on 92.4% of the top websites, but most arrived bundled with a CDN and were never tuned to resist automated access.

Four in 10 sites block AI bots with robots.txt

Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.

75% of the web uses robots.txt - here's how

Three in four of the world's top sites publish a robots.txt, yet few name individual crawlers. A look inside the web's advisory layer: coverage, sophistication and limits.

Revealed: How retailers use tech to block the bots

The web’s big general marketplaces exist to be browsed by everyone. But they are also the second most defended sector online, shutting the door on AI crawlers.

Fashion websites are the hardest to size up

While publishers fight over AI crawlers, fashion has built some of the most heavily defended shop windows on the web. So, which controls are in fashion, in fashion?

The State of Web Access: How different industries behave to bots

Zyte's large-scale audit of web access controls shows how different kinds of businesses exhibit different policies.

The State of Web Access: How today's web raises its barriers

The largest ever audit of web access control mechanisms lifts the lid on how sites are regulating programmatic visits.

Master modern unblocking tactics against the latest anti-bot defenses

Learn how to prepare for modern anti-bot systems with advanced unblocking tactics.

IP Essentials you can't afford to ignore

Advanced Zyte API and Page Object techniques for AI Scraping customization

Residential proxies vs. data center proxies in web scraping

Learn the differences between residential proxies vs. data center proxies in web scraping.