Access handling
Articles from the Zyte blog about Access handling.
What web access actually costs - the tier picture
The barrier headlines sound extreme, but the cost data is more differentiated: 87.9% of top sites are Simple or Easy to access - and that means cost-effective.
The web's layered defences - how access barriers combine
Individual barriers may get the headlines, but it's stacked access control mechanisms that pose the biggest block.
TLS fingerprinting, the handshake that reveals your true intent
TLS fingerprinting fires before you have even asked for a page, and tells you nothing when it does. The least common barrier, and the hardest to diagnose.
JavaScript renders almost half of sites invisible
JavaScript renders 40.6% of top sites unreadable to a plain HTTP request, not by design but as a side effect of how the modern web is built.
Rate limiting is a roadblock, not a speed limit
Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle.
CAPTCHA is more than just a puzzle piece
The famously awkward security mechanism has quietly become an invisible behavioral scoring system. Two vendors enable 89% of CAPTCHAs, and the smallest sites use it most.
Inside antibot, the arms race you can't see
Only 18.5% of top sites run dedicated antibot, but every one of them chose to. Why it's the most intentional barrier in the stack, and the strongest signal of a hardened site.
Firewalls run the web, just not on purpose
Web Application Firewalls run on 92.4% of the top websites, but most arrived bundled with a CDN and were never tuned to resist automated access.
Four in 10 sites block AI bots with robots.txt
Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.
75% of the web uses robots.txt - here's how
Three in four of the world's top sites publish a robots.txt, yet few name individual crawlers. A look inside the web's advisory layer: coverage, sophistication and limits.
Revealed: How retailers use tech to block the bots
The web’s big general marketplaces exist to be browsed by everyone. But they are also the second most defended sector online, shutting the door on AI crawlers.
Fashion websites are the hardest to size up
While publishers fight over AI crawlers, fashion has built some of the most heavily defended shop windows on the web. So, which controls are in fashion, in fashion?