The State of Web Access: How today's web raises its barriers

The open web has a secret: it is not so open anymore.

That shift has been gradual and, to many, pretty invisible - an accumulation of infrastructure choices made by millions of site operators in response to rising threats, rising traffic, and rising competitive pressure.

Bots and automated agents have been crawling the internet for decades, of course. But, as AI-powered crawlers have proliferated and data extraction has industrialized, many websites are changing their posture in response.

Introducing The State of Web Access

To understand how far that hardening has gone, Zyte has conducted the most comprehensive audit of web access barriers ever published.

We analyzed 11,100 of the world's most popular landing pages across 230 countries and 110 industry categories, measuring six distinct access control mechanisms used by each. In total, we analyzed 24,898 websites to deeply understand the patterns in use.

The result is The State of Web Access 2026 - a ground-level survey of what the web actually looks like to the systems trying to read it at scale.

You can browse all the data right now in our complete data explorer. Over the next few weeks, we will be sharing insights on all the juicy findings. Today, here are the top takeaways you need to know.

1. Unmanaged is in the minority

Open access is now a minority position.

Just 18.5% of the landing pages we audited operate without any detectable barrier to automated access.

The other four in five deploy at least one technical barrier - a web application firewall, JavaScript-dependent rendering, a CAPTCHA, rate limiting, TLS fingerprinting, or a dedicated antibot service.

For years, web access has been discussed in terms of exceptions: certain industries are hard to scrape, certain sites fight back. The data shows that a degree of friction is now the default, not the exception.

2. Six barriers, one stack

The six controls we measured each operate differently and demand different responses from any system accessing them.

Web Application Firewalls are the most widely deployed. We found that 63.5% of sites run an identifiable named WAF - one vendor alone accounts for around 35% of all deployments - with a further 29% sitting behind generic WAF signatures. In practice, this means roughly nine in ten popular landing pages have some form of network-layer filtering in place.

• For most sites, the WAF arrives pre-bundled with a CDN and runs at default settings; it is the closest thing the modern web has to a default security posture.
• Just over 40% of landing pages require JavaScript rendering to return meaningful content - a reflection of how thoroughly JavaScript frameworks have colonized web development.
• CAPTCHAs appear on more than one in five pages, with two vendors controlling nearly 90% of deployments.
• Rate limiting and TLS fingerprinting each affect meaningful minorities of sites, and dedicated antibot services - the most sophisticated tier of access control, using behavioral analysis to identify automated traffic - now appear on one in five landing pages.

3. The stacking problem

Individual barriers are well understood. What this research makes newly visible is how they compound.

Half of all landing pages in our dataset deploy two or more barriers simultaneously.

Once a site has a WAF, it is far more likely to have added JavaScript, CAPTCHA, or antibot on top. The extreme end of this spectrum - sites running all six barriers at once - represents just 0.8% of our dataset, but that 0.8% is concentrated precisely in the categories where data is most commercially valuable.

The practical implication is that the difficulty of accessing a site cannot be judged by any single signal. A page that looks easy from a WAF perspective may simultaneously require headless browser infrastructure, CAPTCHA solving, and behavioral fingerprint evasion. Teams building data pipelines need to instrument failures by layer, not by site.

4. How hard is it, really? The cost tier picture

Given the existence of barriers, how cost-effective is it for modern data gatherers to access the sites they need?

Cost is a product of underlying complexity. That's a principle followed by Zyte API, whose five-tier access model scales cost by difficulty - from Simple (plain HTTP, no special handling) through to Advanced (full specialised proxy and evasion configuration). We mapped every domain against this model.

The good news here is, despite the prevalence of access controls, 88% of landing pages still sit at Simple or Easy - the two lowest cost tiers - when accessed through Zyte API.

In other words, The web is not uniformly hard. Complexity concentrates sharply at the high-value end; everything remains cost-efficient to access with the right tooling.

5. robots.txt and the AI crawler standoff

Beyond technical barriers, site operators are increasingly using the web's oldest access protocol to draw new battle lines.

Three-quarters of websites we analyzed employ a robots.txt file to manage access.

Around 13% of sites in our dataset explicitly address AI crawlers in their robots.txt files.

That pattern is most aggressive in content-led industries. Among newspaper sites, 69% have updated their robots.txt to name specific AI agents. Publishing sites stand at 51%, mass media at 44%.

What the data also reveals is that this is a new kind of access dispute.

Search engine crawlers - which have been subject to robots.txt rules for thirty years - are blocked or restricted less frequently than AI crawlers. Publishers appear to have decided that search engines send traffic and AI agents take content. The advisory layer is becoming a negotiating position.

6. Where protection concentrates: fashion leads

Access difficulty is not evenly distributed across the economy. The data shows clearly that barriers cluster where information is commercially valuable, scarce, or tied directly to transaction flow.

Apparel and fashion sites demand the highest mean Zyte API access tier of any industry we measured - a reflection of how aggressively the sector defends pricing, inventory, and trend data against competitive scraping.

Consumer electronics, real estate, and travel follow closely.

At the other end of the spectrum, public sector, agriculture, and media and publishing sites remain comparatively open - sectors where discoverability matters more than data exclusivity.

Most defended (mean recommended Zyte API tier):

Least defended (mean recommended Zyte API tier):

Zyte API tiers: 1 - Simple (plain HTTP), 2 - Easy (JavaScript rendering required), 3 - Moderate (browser with basic fingerprint handling), 4 - Complex (residential proxies or advanced evasion), 5 - Advanced (full specialised infrastructure). A mean above 2.0 means the average site in that sector requires at minimum a headless browser. Fleet mean: 1.58.

What this means

The State of Web Access 2026 is the first large-scale measurement of how the web's access layer has actually been built.

The picture it paints is of a web that has made a structural choice: data access is no longer assumed. It is priced, filtered, and increasingly contested.

The full research - covering all six barriers, 230 countries, 110 industries, and the complete vendor landscape - is available at zyte.com/sowa/2026.