Overview
CAPTCHA and JavaScript define Newspapers' active barrier layer
| Value | |
|---|---|
| WAF | 96% |
| Antibot | 8% |
| CAPTCHA | 21% |
| JavaScript | 36% |
| Rate Limiting | 15% |
| TLS | 10% |
WAF coverage at 96% is near-universal. JavaScript rendering (36%) and CAPTCHA (21%) are the sector's defining active barriers, reflecting the value publishers place on controlling how their content is accessed. Antibot tools feature in only 8% of sites — publishers prioritise access policy via robots.txt over technical bot fingerprinting.
Barrier Landscape
JavaScript tops Newspapers' active secondary defences
| Value | |
|---|---|
| WAF | 96% |
| Antibot | 8% |
| CAPTCHA | 21% |
| JavaScript | 36% |
| Rate Limiting | 15% |
| TLS | 10% |
| Has robots.txt | 90% |
| Blocks AI crawlers | 64% |
WAF is deployed by 96% of Newspaper sites. JavaScript rendering leads active secondary barriers at 36%, followed by CAPTCHA at 21% and Rate Limiting at 15%. Nine in ten sites publish a robots.txt file and 64% use it to block AI crawlers — a policy commitment that far outpaces technical barrier deployment in this sector.
Single-barrier sites are Newspapers' most common configuration
| % of sites | |
|---|---|
| 0 barriers | 2% |
| 1 barrier | 45% |
| 2 barriers | 27% |
| 3 barriers | 20% |
| 4 barriers | 3% |
| 5 barriers | 3% |
| 6 barriers | 0% |
45% of Newspaper sites carry exactly one barrier — almost always WAF operating alone. Two-barrier configurations (27%) typically pair WAF with JavaScript rendering. Just 6% of sites reach four or more barriers, concentrated among the largest national titles with commercially sensitive content.
Nearly half of Newspaper sites are Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.77 |
A mean recommended tier of 1.77 and a median of Easy place Newspapers around the middle of the sectors surveyed. 46% of sites are Simple to access; 17% require Moderate or above.
Most Newspaper sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 46% |
| 2 - Easy | 37% |
| 3 - Moderate | 12% |
| 4 - Complex | 4% |
| 5 - Advanced | 1% |
Simple (46%) is the most common tier. Simple and Easy together account for 83% of sites, while 12% land at Moderate and 5% at Complex or Advanced.
robots.txt
Nine in ten Newspaper sites publish a robots.txt
| % of sites | |
|---|---|
| Publish a valid robots.txt | 90% |
robots.txt adoption in Newspapers is among the highest of any sector surveyed, with 90% of sites publishing a valid file. Just 10% leave crawl policy undefined — a significantly lower rate than the global average of around 24%.
Half of all Newspaper sites name 16 or more bots
| % of sites | |
|---|---|
| 0 | 11% |
| 1 | 13% |
| 2–4 | 9% |
| 5–9 | 9% |
| 10–15 | 8% |
| 16+ | 50% |
The median Newspaper site names 15 distinct bots in its robots.txt and the mean of 33.8 reflects the long tail of sites maintaining exhaustive bot lists. Half of all Newspaper sites name 16 or more bots by name. This level of specificity is rare across sectors and reflects deliberate, actively maintained crawl policy in newsrooms.
AI crawlers top Newspaper bot categories in robots.txt
| % of sites | |
|---|---|
| AI crawlers | 66% |
| SEO / search indexing | 29% |
| SEO audit tools | 40% |
| Commercial scrapers | 39% |
| Social / content scrapers | 10% |
| Archive / research crawlers | 21% |
| Monitoring / uptime bots | 2% |
66% of Newspaper sites name at least one AI crawler in their robots.txt — the most mentioned bot category in the sector. SEO audit tools (40%) and commercial scrapers (39%) follow closely. Figures are percentages of all 100 Newspaper sites, including the 10% with no robots.txt at all.
Uniform and path-mixed rules split Newspaper robots.txt files evenly
| % of sites | |
|---|---|
| Uniform per-agent rules | 41% |
| Path-level mixed rules | 42% |
| Allow-only (no Disallow rules) | 0% |
| Mixed AI posture | 13% |
| Blanket Disallow: / | 3% |
Of the 90% of Newspaper sites with a valid robots.txt, posture splits roughly evenly between uniform per-agent rules (41%) and path-level mixed rules (42%). A further 13% maintain a mixed AI posture — blocking some AI crawlers while explicitly allowing others. Only 3% issue a blanket Disallow: /. Figures are percentages of all 100 sites.
GPTBot and CCBot face the most Disallow rules in Newspaper robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 86% | 36% |
| GPTBot | 51% | 3% |
| Google-Extended | 46% | 6% |
| CCBot | 51% | 1% |
| PerplexityBot | 38% | 6% |
| anthropic-ai | 42% | 2% |
| ClaudeBot | 41% | 2% |
| ChatGPT-User | 34% | 8% |
| Claude-Web | 36% | 3% |
| Bytespider | 36% | 2% |
| OAI-SearchBot | 25% | 12% |
| cohere-ai | 34% | 2% |
| Applebot-Extended | 33% | 2% |
| Amazonbot | 29% | 3% |
| Diffbot | 30% | 1% |
| FacebookBot | 27% | 2% |
| omgilibot | 27% | 1% |
| Meta-ExternalAgent | 25% | 2% |
| omgili | 26% | 1% |
| YouBot | 22% | 2% |
| PetalBot | 20% | 1% |
| Timpibot | 20% | 1% |
| TurnitinBot | 20% | — |
| Scrapy | 18% | 1% |
| ImagesiftBot | 18% | 1% |
Values are percentages of all 100 Newspaper sites surveyed. The wildcard (*) at 86% Disallow reflects a default-restrict posture applied to uncategorised crawlers. Among named bots, GPTBot and CCBot lead blocks at 51% each. OAI-SearchBot (12% Allow) and ChatGPT-User (8% Allow) attract the most explicit whitelisting — a pattern of restricting training crawlers while selectively permitting search-facing agents.
Industry Cohort
Newspapers lead their cohort on AI blocking, not on technical barriers
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Advertising & Marketing | 97% | 23% | 32% | 31% | 21% | 14% | 87% | 6% | Tier 1 |
| Mass Media | 98% | 12% | 21% | 49% | 16% | 8% | 92% | 43% | Tier 2 |
| Newspapers | 96% | 8% | 21% | 36% | 15% | 10% | 90% | 64% | Tier 2 |
| Public Relations | 95% | 10% | 14% | 25% | 14% | 14% | 76% | 6% | Tier 1 |
| Publishing | 95% | 16% | 24% | 45% | 21% | 11% | 92% | 49% | Tier 2 |
| Writing & Editing | 98% | 14% | 33% | 31% | 11% | 6% | 79% | 11% | Tier 1 |
Newspapers' 64% AI-blocking rate is the highest in the Media & Publishing cohort — around 15 points above Publishing, the next-highest sub-category. JavaScript adoption (36%) also leads the group. Antibot (8%) and TLS (10%) sit at or near the cohort floor, consistent with a sub-category that favours robots.txt policy over middleware deployment.