Overview
JavaScript is the UK's most common secondary barrier
| Value | |
|---|---|
| WAF | 93.4% |
| Antibot | 21.5% |
| CAPTCHA | 22.1% |
| JavaScript | 42.7% |
| Rate Limiting | 13.1% |
| TLS | 18.4% |
The UK's barrier footprint is concentrated in WAF coverage and JavaScript-rendered pages. Antibot, CAPTCHA, Rate Limiting, and TLS are deployed selectively — typically layered on top of WAF in sectors with higher-value or more commercially sensitive data.
Barrier Landscape
JavaScript tops the UK's active secondary defences
| Value | |
|---|---|
| WAF | 93.4% |
| Antibot | 21.5% |
| CAPTCHA | 22.1% |
| JavaScript | 42.7% |
| Rate Limiting | 13.1% |
| TLS | 18.4% |
| Has robots.txt | 81.6% |
| Blocks AI crawlers | 14.3% |
WAF adoption is near-universal and broadly in line with the global average of 93.1%. JavaScript rendering is the barrier that most frequently differentiates UK sites from a simple HTTP scrape. The near-parity of antibot and CAPTCHA rates suggests that bot-management vendors here frequently bundle CAPTCHA as a fallback trigger rather than deploying it as a standalone layer.
Two in three UK sites carry two or more barriers
| % of sites | |
|---|---|
| 0 barriers | 4.2% |
| 1 barrier | 29.7% |
| 2 barriers | 35% |
| 3 barriers | 17.6% |
| 4 barriers | 9% |
| 5 barriers | 4.1% |
| 6 barriers | 0.4% |
Only 4.2% of UK sites present no detectable barriers. The modal outcome is two barriers — most commonly WAF combined with JavaScript rendering. Three-barrier sites represent nearly one in five, typically adding either antibot or CAPTCHA to that pair.
UK sites sit comfortably in the Easy access band
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
A mean of 1.66 places the UK firmly below tier 2, reflecting the large share (47.8%) of sites accessible via plain HTTP. The skew toward the lower end means the UK is one of the more accessible major web markets — only the ~11% of sites requiring tiers 3–5 materially pull the mean upward.
Nearly nine in ten UK sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 47.8% |
| 2 - Easy | 41.1% |
| 3 - Moderate | 8.6% |
| 4 - Complex | 1.7% |
| 5 - Advanced | 0.7% |
The UK's tier distribution sits close to the global average. The 47.8% at tier 1 suggests a significant share of sites that deploy WAF rely on CDN-bundled rules that do not materially impede structured HTTP requests — the WAF is present but passive at tier 1.
robots.txt
AI crawlers top the UK's robots.txt bot categories
| % of sites | |
|---|---|
| AI crawlers | 15.2% |
| SEO / search indexing | 10.3% |
| SEO audit tools | 9.6% |
| Commercial scrapers | 7.1% |
| Social / content scrapers | 6.2% |
| Archive / research crawlers | 4.7% |
| Monitoring / uptime bots | 1.4% |
15% of all UK sites surveyed name at least one AI crawler in their robots.txt — more than any other bot category. Figures are percentages of all 1,111 UK sites, so the base includes sites with no robots.txt at all. Archive and monitoring bots remain rarely addressed, suggesting these categories are perceived as lower risk or lower relevance by UK webmasters.
Path-mixed rules are the UK's most common robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 48.1% |
| Path-level mixed rules for same agent | 33.6% |
| Allow-only (no disallow rules) | 2% |
| Mixed AI posture (block some, allow others) | 2.4% |
| Blanket Disallow: / | 2.6% |
Figures are percentages of all 1,111 UK sites surveyed — the 18% with no robots.txt are implicit in the gap to 100%. Of the 82% that do publish one, posture splits roughly 48/34: uniform per-agent rules vs path-level mixing. The bottom three rows are independent signals: 2% publish an allow-only file, 2.4% maintain a mixed AI posture, and 2.6% issue a blanket Disallow: /.
GPTBot faces the most Disallow rules in UK robots.txt files
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 12% | 1.9% |
| CCBot | 9.9% | 1.1% |
| ClaudeBot | 8.9% | 0.9% |
| Google-Extended | 8.4% | 1.4% |
| Bytespider | 6.8% | 0.4% |
| meta-externalagent | 5.3% | 0.5% |
| ChatGPT-User | 6.1% | 2.3% |
| PerplexityBot | 5.9% | 1.8% |
| Applebot-Extended | 5% | 0.4% |
| OAI-SearchBot | 4.1% | 2.1% |
| anthropic-ai | 5.4% | 0.7% |
Values are percentages of all 1,111 UK sites surveyed (not just those with robots.txt). GPTBot leads disallowance at 12%, mirroring its global prominence. The divergence between ChatGPT-User's 2.3% allow rate and its 6.1% block rate suggests a meaningful cohort of UK publishers who explicitly whitelist the browsing agent while still blocking the training crawler.
UK Newspaper sites all name at least one bot to restrict
| % of sites | |
|---|---|
| Newspapers | 90% |
| Publishing | 90% |
| Entertainment | 80% |
| Airlines | 70% |
| Foreign Language | 70% |
| Healthcare | 70% |
| Mass Media | 70% |
| Online Services | 70% |
| Retail | 70% |
| Comics & Animation | 60% |
| Crafts | 60% |
| Food & Beverages | 60% |
| Hospitality | 60% |
| Textiles & Nonwovens | 60% |
| Travel & Tourism | 60% |
Content-producing sectors dominate: newspapers and publishers both hit 90%, with entertainment close behind at 80%. Airlines appear at joint fourth — a UK-specific pattern driven by fare scraping and dynamic pricing concerns. The 60% cluster spanning Crafts, Food, Hospitality, Textiles, and Travel reflects the long tail of industries that have adopted template-driven robots.txt files naming common SEO and content crawlers.
UK Real Estate sites lead in explicitly allowing crawlers
| % of sites | |
|---|---|
| Real Estate | 60% |
| Airlines | 50% |
| Photography | 50% |
| Publishing | 50% |
| Travel & Tourism | 50% |
| Education | 40% |
| Social & Charitable | 40% |
| Software & Dev | 40% |
| Telecom | 40% |
| Accounting & Auditing | 30% |
| Automotive | 30% |
| Biotech | 30% |
| Computer Hardware | 30% |
| Construction & Maintenance | 30% |
| Design | 30% |
Real estate leads by a clear margin: property portals and estate agents benefit commercially from being indexed by aggregators, price comparison tools, and lead-generation crawlers, so explicit whitelisting is a deliberate business decision. Airlines and Travel & Tourism follow at 50%, consistent with the distribution model of the aviation and hospitality industry. Publishing's joint second placement is notable — the same sector that leads named-bot blocking also leads explicit whitelisting, reflecting a nuanced two-tier strategy: block training crawlers, welcome discovery bots.
Industry Patterns
Real estate and luxury goods are the UK's hardest industries to access
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Real Estate | 100% | 30% | 30% | 70% | 60% | 10% | 60% | 30% | Tier 3 |
| Jewellery & Luxury | 100% | 20% | 20% | 30% | 40% | 60% | 60% | 0% | Tier 3 |
| Human Resources | 100% | 80% | 60% | 50% | 40% | 20% | 70% | 10% | Tier 2 |
| Restaurants | 80% | 30% | 20% | 70% | 40% | 20% | 70% | 10% | Tier 2 |
| Medical Devices | 100% | 0% | 0% | 40% | 10% | 0% | 100% | 0% | Tier 2 |
| Food & Beverages | 100% | 60% | 30% | 70% | 40% | 0% | 90% | 50% | Tier 2 |
| Music | 100% | 40% | 30% | 70% | 30% | 20% | 80% | 10% | Tier 2 |
| Reference | 90% | 60% | 30% | 50% | 30% | 10% | 60% | 40% | Tier 2 |
| Photography | 90% | 30% | 50% | 20% | 20% | 90% | 80% | 30% | Tier 2 |
| Adult | 100% | 40% | 40% | 70% | 10% | 20% | 90% | 10% | Tier 2 |
| Advertising & Mktg | 100% | 30% | 50% | 60% | 0% | 30% | 90% | 10% | Tier 2 |
| Machinery | 90% | 0% | 0% | 60% | 10% | 0% | 80% | 0% | Tier 2 |
| Printing | 100% | 0% | 10% | 20% | 0% | 0% | 90% | 0% | Tier 2 |
| Distance Learning | 100% | 40% | 50% | 50% | 30% | 10% | 70% | 30% | Tier 2 |
| Pharmaceuticals | 100% | 30% | 40% | 50% | 40% | 20% | 70% | 0% | Tier 2 |
Property portals and luxury retailers sit at the top of the difficulty scale — both sectors average tier 2.7, driven by heavy JavaScript rendering combined with rate limiting and TLS fingerprinting. Human resources and restaurants follow, where structured data has high commercial value. Medical devices is a notable outlier: near-zero antibot and CAPTCHA, yet a mean tier of 2.1 — difficulty comes from JavaScript complexity, not deliberate bot defence.
Civic, professional, and creative sectors are the UK's most accessible industries programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Construction | 80% | 0% | 10% | 30% | 0% | 0% | 80% | 10% | Tier 2 |
| Biotech | 90% | 30% | 50% | 60% | 10% | 20% | 80% | 30% | Tier 2 |
| Government | 80% | 20% | 0% | 20% | 10% | 10% | 70% | 0% | Tier 2 |
| Legal | 90% | 0% | 10% | 20% | 10% | 10% | 90% | 0% | Tier 2 |
| Training & Cert. | 90% | 20% | 10% | 20% | 10% | 10% | 90% | 20% | Tier 1 |
| Gambling | 90% | 30% | 30% | 70% | 50% | 10% | 50% | 0% | Tier 1 |
| Finance | 80% | 20% | 0% | 20% | 10% | 10% | 70% | 20% | Tier 1 |
| Museums | 100% | 10% | 10% | 30% | 0% | 0% | 70% | 0% | Tier 1 |
| Public Relations | 100% | 20% | 0% | 10% | 0% | 40% | 100% | 10% | Tier 1 |
| Market Research | 64% | 9% | 9% | 36% | 0% | 9% | 64% | 0% | Tier 1 |
| Facilities Services | 90% | 10% | 20% | 20% | 0% | 10% | 60% | 0% | Tier 1 |
| Public Utility | 100% | 0% | 0% | 50% | 0% | 0% | 90% | 0% | Tier 1 |
| Graphic Design | 100% | 40% | 40% | 40% | 20% | 50% | 100% | 20% | Tier 1 |
| Political Orgs | 80% | 0% | 0% | 20% | 0% | 10% | 80% | 0% | Tier 1 |
| Recreational Facilities | 100% | 10% | 0% | 60% | 0% | 0% | 80% | 10% | Tier 1 |
Recreational facilities is the most accessible industry in the UK sample — a mean tier of 1.0 means every site returns a clean HTTP response. Graphic design, political organisations, and public utility follow at 1.1–1.2. WAF coverage remains high across the board, but these sectors have not layered behavioural defences on top of it. Notably, graphic design carries moderate barrier counts yet stays at tier 1 — its defences are present but ineffective against automated access.