Overview
Reference's WAF coverage is unusually low — active barriers carry the load
| Value | |
|---|---|
| WAF | 81% |
| Antibot | 15% |
| CAPTCHA | 17% |
| JavaScript | 31% |
| Rate Limiting | 16% |
| TLS | 3% |
WAF covers only 81% of Reference sites, well below the global norm. JavaScript (31%) and CAPTCHA (17%) absorb the defensive weight in a sector that leans on active challenge layers over firewall coverage.
Barrier Landscape
WAF leads Reference at 81% — JavaScript is the main secondary layer
| Value | |
|---|---|
| WAF | 81% |
| Antibot | 15% |
| CAPTCHA | 17% |
| JavaScript | 31% |
| Rate Limiting | 16% |
| TLS | 3% |
| Has robots.txt | 60% |
| Blocks AI crawlers | 17% |
WAF dominates at 81%. JavaScript (31%) leads active secondary barriers, followed by CAPTCHA (17%). The remaining three barriers all sit at or below 16% — a concentrated rather than broadly stacked profile.
Reference's barrier counts are spread — no single configuration dominates
| % of sites | |
|---|---|
| 0 barriers | 18% |
| 1 barrier | 35% |
| 2 barriers | 26% |
| 3 barriers | 13% |
| 4 barriers | 3% |
| 5 barriers | 5% |
| 6 barriers | 0% |
Barrier counts vary widely across Reference sites. Single-barrier configurations are the modal count at 35%, but the distribution is broad. 21% of sites carry three or more.
Simple is the dominant access tier across Reference
| Mean recommended tier | |
|---|---|
| All sites | 1.50 |
A mean recommended tier of 1.50 and a median of Simple place Reference toward the easier end of the sectors surveyed. 66% of sites are Simple to access; 9% require Moderate or above.
Simple is the dominant tier in Reference, but Easy accounts for a significant share
| % of sites | |
|---|---|
| 1 - Simple | 66% |
| 2 - Easy | 25% |
| 3 - Moderate | 3% |
| 4 - Complex | 5% |
| 5 - Advanced | 1% |
Simple (66%) is the most common tier. Simple and Easy together account for 91% of sites, while 3% land at Moderate and 6% at Complex or Advanced.
robots.txt
Nearly two-thirds of Reference sites publish any robots.txt
| % of sites | |
|---|---|
| Publish a valid robots.txt | 60% |
Only 60% of Reference sites have a valid robots.txt. Nearly 40% leave crawl policy undefined — one of the lower coverage rates across surveyed sectors.
Most Reference sites name one bot or none in their robots.txt
| % of sites | |
|---|---|
| 0 | 48% |
| 1 | 21% |
| 2–4 | 10% |
| 5–9 | 7% |
| 10–15 | 6% |
| 16+ | 8% |
48% of Reference sites name zero agents and 21% name one. A mean of 6.2 is pulled up by a small number of sites with extensive policies, while the median stays at 1.
AI crawlers draw the most robots.txt attention in Reference
| % of sites | |
|---|---|
| AI crawlers | 20% |
| SEO / search indexing | 12% |
| SEO audit tools | 16% |
| Commercial scrapers | 8% |
| Social / content scrapers | 2% |
| Archive / research crawlers | 4% |
| Monitoring / uptime bots | 2% |
AI crawlers are named in 20% of Reference robots.txt files, followed by SEO audit tools at 16%. All figures are percentages of all 100 sites including those with no robots.txt.
Uniform per-agent rules dominate Reference robots.txt files
| % of sites | |
|---|---|
| Uniform per-agent rules | 39% |
| Path-level mixed rules | 15% |
| Allow-only (no Disallow rules) | 4% |
| Mixed AI posture | 1% |
| Blanket Disallow: / | 2% |
Uniform per-agent rules lead at 39%, with path-level mixed rules at 15%. Mixed AI posture (1%) and blanket Disallow (2%) are minimal.
GPTBot and MJ12bot lead named Disallow rules in Reference robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 42% | 21% |
| GPTBot | 13% | 1% |
| MJ12bot | 13% | 1% |
| Amazonbot | 10% | — |
| ClaudeBot | 9% | 1% |
| SemrushBot | 8% | 1% |
| Baiduspider | 4% | 3% |
| Bytespider | 7% | — |
| CCBot | 7% | — |
| Yandex | 6% | 1% |
| AhrefsBot | 6% | 1% |
| Applebot | 4% | 2% |
| PetalBot | 5% | 1% |
| ChatGPT-User | 3% | 2% |
| dotbot | 5% | — |
| Bingbot | 3% | 2% |
| Googlebot | 2% | 3% |
| Slurp | 4% | 1% |
| YandexBot | 4% | 1% |
| Diffbot | 4% | — |
| Google-Extended | 3% | 1% |
| OAI-SearchBot | 2% | 2% |
| Googlebot-Image | 2% | 2% |
| BLEXBot | 4% | — |
| psbot | 4% | — |
Values are percentages of all 100 Reference sites surveyed. The wildcard (*) leads at 42% Disallow. Among named agents, GPTBot and MJ12bot share the top Disallow position, each named by 13% of sites.
Industry Cohort
Reference has the lowest WAF coverage in the Education cohort
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Distance Learning | 97% | 27% | 37% | 43% | 26% | 16% | 85% | 25% | Tier 2 |
| Education | 98% | 17% | 18% | 37% | 18% | 10% | 81% | 15% | Tier 2 |
| Foreign Language | 96% | 15% | 20% | 42% | 14% | 13% | 82% | 6% | Tier 1 |
| Libraries | 92% | 16% | 9% | 36% | 16% | 11% | 69% | 8% | Tier 1 |
| Reference | 81% | 15% | 17% | 31% | 16% | 3% | 60% | 17% | Tier 1 |
| Science | 90% | 15% | 11% | 35% | 15% | 9% | 79% | 10% | Tier 2 |
| Training & Certification | 93% | 15% | 19% | 41% | 17% | 11% | 75% | 7% | Tier 1 |
WAF at 81% is the lowest in the Education cohort. JavaScript (31%) compensates as the leading active barrier. Tier difficulty (mean 1.43) sits below the cohort median.