75% of the web uses robots.txt - here's how

In 1994, a Dutch software developer named Martijn Koster published a brief proposal to a web standards mailing list.

Web crawlers were beginning to index the internet, and some site operators wanted a way to communicate which parts of their site they preferred crawlers to leave alone. The proposal was a simple text file - robots.txt - placed at the root of a domain, containing instructions any well-behaved crawler could read and follow.

The robots exclusion protocol, as it became known, was historically considered an informal request rather than a technical barrier to scraping. It could not block a crawler that chose to ignore it. It was, in Koster's own framing, a "notice to robots" - the web equivalent of a sign on a gate rather than a lock on it.

The state of robots.txt

Thirty years later, 75.8% of the world's 11,100 most popular landing pages publish a robots.txt file, according to Zyte's State of Web Access 2026 research - a higher baseline than any of the six access control mechanisms measured.

And the instructions in those files have become considerably more elaborate than anything Koster envisioned.

The protocol that took 28 years to reach formal RFC status (RFC 9309, published in 2022) has quietly become the web's primary instrument for declaring access policy.

Understanding what those declarations actually say - and what they can and cannot do - is the starting point for understanding how the web's access layer is governed.

From basic to expert robots.txt files

Not every robots.txt file is doing the same thing.

Among the 8,419 sites in our dataset with a valid robots.txt, we found a wide range of sophistication.

Basic: At one end, 11.2% have published nothing beyond a wildcard catch-all: User-agent: * followed by a blanket Disallow or Allow. This is a configuration - often a CMS default - that says something about automated access in general but nothing about any specific crawler.

Expert: At the other end, 8.5% have reached what we categorise as Expert level: files that actively manage access from both ends simultaneously, specifying what different crawlers can and cannot do at the path level, in some cases across hundreds of named agents.

Advanced: Between those poles, 56.8% have made what we call an Advanced choice - a one-sided access decision, either a comprehensive whitelist or a blacklist, but not both.

The striking finding here is that 65% of sites with robots.txt have moved beyond the default. That is not what you'd expect from a 30-year-old file format that most developers encounter once and forget. It suggests something closer to active policy maintenance.

A third of sites target specific agents

The population that has made the most deliberate choices is a specific subset: 30.7% of all sites - 3,410 in the dataset - name at least one specific User-agent by string rather than relying on the wildcard.

Among those that do name agents, granularity varies enormously. Most name fewer than five. But 9.7% of naming sites address ten or more distinct crawlers. The most elaborate file in the dataset names 1,774 separate agents - a list so comprehensive it almost certainly reflects professional curation rather than organic accumulation.

Industries that depend on controlling how their content is distributed lead on named-agent specificity.

Publishing sites exert most active control

Newspapers are the clear frontrunner: 66% of newspaper sites explicitly name five or more distinct crawlers by User-agent. Mass media and publishing cluster just behind at 48% each.

These are sectors where the specific identity of the crawler - not just whether it is automated - carries commercial and legal significance.

At the other end, the sectors least likely to name specific agents are those whose business models depend on discoverability: logistics, public services, agriculture. Blocking automated access is actively contrary to their interests, so a permissive wildcard is a rational choice.

robots.txt delays crawls by 10 seconds

Of the 11,100 sites in the dataset, 1,373 deploy a Crawl-delay directive - a field that tells crawlers how many seconds to wait between requests.

The distribution of those delay values tells its own story.

• The modal value, used by 44.6% of delay-setting sites, is exactly 10 seconds.
• But the 10-second spike is almost certainly not the result of 600 independent site operators independently calculating that 10 seconds is the optimal crawl interval.
• It likely reflects a copied CMS default - Drupal, WordPress, and other platforms have shipped with Crawl-delay: 10 as a template value for years, and operators who have never thought carefully about crawl budgeting have left it in place.

At 30 seconds and above, comprehensive crawling within any commercial timeframe becomes impractical. Sites at this threshold are effectively setting a passive exclusion: not blocking access in any legal sense, but making systematic indexing economically unviable for most use cases.

SEO audit tools draw the longest delays

When you look at which specific agents receive the longest crawl delays, the result is not what you might expect given the current debate about AI crawlers.

SEO audit tools are throttled hardest - by a significant margin. These are commercial link auditing tools that index pages to build backlink databases, and operators who have taken the time to name them specifically have decided they are not welcome to crawl freely.

AI crawlers sit near the bottom: ClaudeBot and Bytespider average 16 seconds, CCBot 15 seconds, GPTBot 14 seconds. The delay is real but modest - inconvenienced rather than neutralised.

Operators who set crawl delays for AI agents have calibrated something different from the operators who throttle SEO tools to near-impracticality. Whether that reflects a different threat assessment or simply the fact that AI-specific delay-setting is newer, the delay data and the Disallow data tell different stories.

Rule architecture: more intentional than it looks

The path-level structure of robots.txt rules across the dataset reveals a level of intentionality that aggregate statistics tend to obscure.

• 45.9% of all sites use uniform per-agent rules - consistent allow or block without path exceptions.
• But 29.9% mix allows and blocks for the same bot at path level: a crawler may be blocked from /admin/, /checkout/, or /internal/ while remaining welcome on /blog/ or /sitemap.xml/. This kind of rule requires deliberate path-level thinking about which sections of a site carry sensitive data and which are meant to be publicly indexed.
• A further 3.1% publish no Disallow rules at all - an explicit open posture that formally welcomes all well-behaved crawlers. This is not an absence of robots.txt; it is a positive statement. The 1.4% with a blanket Disallow: / - blocking all access to all paths - are at the opposite pole.

Where a robots.txt rule loses its force

Unlike the technical barriers measured elsewhere in State of Web Access 2026 - WAFs, antibot systems, rate limiting, TLS fingerprinting - robots.txt does not physically stop a crawler.

A Disallow rule is a declaration of preference, not an obstacle. The file can only be acted on by a crawler that chooses to read and respect it.

But the legal weight attached to that notice is no longer as thin as it once was. The EU AI Act's General Purpose AI Code of Conduct requires signatory GPAI providers to respect robots.txt as a machine-readable opt-out - a binding commitment for those who have signed. For AI training data, EU law is establishing a requirement to respect machine-readable opt-outs, even without naming robots.txt specifically.

The acceleration of robots.txt specificity in recent years - particularly around AI crawlers - reflects this shift. Site operators are writing detailed, sophisticated access policies in a format whose enforceability is actively evolving. The humble text file is doing more policy work than it was ever designed to handle - and the legal infrastructure around it is beginning to catch up.