Overview
JavaScript and Antibot both elevate Adult's active barrier profile
| Value | |
|---|---|
| WAF | 98% |
| Antibot | 36% |
| CAPTCHA | 33% |
| JavaScript | 36% |
| Rate Limiting | 34% |
| TLS | 20% |
WAF leads at 98%. JavaScript (36%) and Antibot (36%) together form a multi-layer active stack — more than most sectors deploy alongside WAF.
Barrier Landscape
WAF leads Adult at 98% — JavaScript is the main secondary layer
| Value | |
|---|---|
| WAF | 98% |
| Antibot | 36% |
| CAPTCHA | 33% |
| JavaScript | 36% |
| Rate Limiting | 34% |
| TLS | 20% |
| Has robots.txt | 85% |
| Blocks AI crawlers | 9% |
WAF dominates at 98%. JavaScript and Antibot are tied as the leading secondary barriers at 36% each. The remaining three barriers all sit at or below 34% — a concentrated rather than broadly stacked profile.
Adult's barrier counts are spread — no single configuration dominates
| % of sites | |
|---|---|
| 0 barriers | 2% |
| 1 barrier | 36% |
| 2 barriers | 23% |
| 3 barriers | 9% |
| 4 barriers | 5% |
| 5 barriers | 22% |
| 6 barriers | 3% |
Barrier counts vary widely across Adult sites. Single-barrier configurations are the modal count at 36%, but the distribution is broad. 39% of sites carry three or more.
Adult is predominantly Simple to access — with a significant Easy minority
| Mean recommended tier | |
|---|---|
| All sites | 1.51 |
A mean recommended tier of 1.51 and a median of Simple place Adult toward the easier end of the sectors surveyed. 55% of sites are Simple to access; 4% require Moderate or above.
Simple is the dominant tier in Adult, but Easy accounts for a significant share
| % of sites | |
|---|---|
| 1 - Simple | 55% |
| 2 - Easy | 41% |
| 3 - Moderate | 2% |
| 4 - Complex | 2% |
| 5 - Advanced | 0% |
Simple (55%) is the most common tier. Simple and Easy together account for 96% of sites, while 2% land at Moderate and 2% at Complex or Advanced.
robots.txt
Most Adult sites publish a robots.txt — coverage above the norm
| % of sites | |
|---|---|
| Publish a valid robots.txt | 85% |
85% of Adult sites publish a valid robots.txt. The 15% without a file leave crawl policy undefined — a lower gap than the cross-sector average of around 24%.
Adult sites name a handful of agents in robots.txt — few commit to comprehensive policies
| % of sites | |
|---|---|
| 0 | 27% |
| 1 | 38% |
| 2–4 | 22% |
| 5–9 | 8% |
| 10–15 | 2% |
| 16+ | 3% |
The mean of 2.5 named agents reflects limited bot-policy investment in Adult. 27% of sites name no agents; 38% name exactly one. Only 3% maintain a list of 16 or more named bots.
SEO crawlers draw the most robots.txt attention in Adult
| % of sites | |
|---|---|
| AI crawlers | 9% |
| SEO / search indexing | 25% |
| SEO audit tools | 8% |
| Commercial scrapers | 1% |
| Social / content scrapers | 1% |
| Archive / research crawlers | 19% |
| Monitoring / uptime bots | 0% |
SEO / search indexing bots are mentioned by 25% of Adult sites. AI crawlers follow at 9%. All figures are percentages of all 100 Adult sites surveyed.
Uniform per-agent rules dominate Adult robots.txt files
| % of sites | |
|---|---|
| Uniform per-agent rules | 56% |
| Path-level mixed rules | 25% |
| Allow-only (no Disallow rules) | 4% |
| Mixed AI posture | 2% |
| Blanket Disallow: / | 0% |
Uniform per-agent rules lead at 56%, with path-level mixed rules at 25%. Mixed AI posture (2%) and blanket Disallow (0%) are minimal.
ia_archiver leads named Disallow rules in Adult robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 65% | 29% |
| ia_archiver | 18% | — |
| CCBot | 9% | 2% |
| Amazonbot | 8% | 2% |
| Bytespider | 8% | 2% |
| Googlebot | 5% | 5% |
| ClaudeBot | 8% | 1% |
| GPTBot | 7% | 1% |
| Applebot-Extended | 6% | 1% |
| Yandex | 5% | 1% |
| Anthropic-ai | 3% | 2% |
| AwarioRssBot | 3% | 2% |
| AwarioSmartBot | 3% | 2% |
| Cohere-ai | 3% | 2% |
| DataForSeoBot | 3% | 2% |
| ImagesiftBot | 3% | 2% |
| Magpie-crawler | 3% | 2% |
| Omgili | 3% | 2% |
| Omgilibot | 3% | 2% |
| Peer39_crawler | 3% | 2% |
| Peer39_crawler/1.0 | 3% | 2% |
| YouBot | 3% | 2% |
| AhrefsBot | 4% | 1% |
| PerplexityBot | 4% | 1% |
| Google-Extended | 5% | — |
Values are percentages of all 100 Adult sites surveyed. The wildcard (*) leads at 65% Disallow. Among named agents, ia_archiver leads the Disallow list at 18%, ahead of CCBot at 9%.
Industry Cohort
Adult is the most accessible sub-category in the Entertainment cohort
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Adult | 98% | 36% | 33% | 36% | 34% | 20% | 85% | 9% | Tier 1 |
| Entertainment | 92% | 17% | 18% | 46% | 23% | 13% | 77% | 30% | Tier 2 |
| Gambling | 95% | 31% | 24% | 53% | 35% | 14% | 57% | 6% | Tier 1 |
| Music | 96% | 21% | 19% | 52% | 26% | 12% | 84% | 25% | Tier 2 |
| TV & Movies | 99% | 30% | 25% | 59% | 33% | 13% | 70% | 11% | Tier 1 |
A median tier of Simple and mean of 1.50 make Adult the easiest to access programmatically within Entertainment. WAF (98%) and AI blocking (9%) are in line with cohort peers.