Overview
CAPTCHA leads Advertising & Marketing's secondary barrier stack above the norm
| Value | |
|---|---|
| WAF | 97% |
| Antibot | 23% |
| CAPTCHA | 32% |
| JavaScript | 31% |
| Rate Limiting | 21% |
| TLS | 14% |
WAF at 97% is the dominant barrier. CAPTCHA (32%) leads secondary coverage. The remaining barriers sit materially lower, giving Advertising & Marketing a narrow but defined secondary layer.
Barrier Landscape
WAF leads Advertising & Marketing at 97% — CAPTCHA is the main secondary layer
| Value | |
|---|---|
| WAF | 97% |
| Antibot | 23% |
| CAPTCHA | 32% |
| JavaScript | 31% |
| Rate Limiting | 21% |
| TLS | 14% |
| Has robots.txt | 87% |
| Blocks AI crawlers | 6% |
WAF dominates at 97%. CAPTCHA (32%) leads active secondary barriers, followed by JavaScript (31%). The remaining three barriers all sit at or below 23% — a concentrated rather than broadly stacked profile.
Advertising & Marketing's barrier counts are spread — no single configuration dominates
| % of sites | |
|---|---|
| 0 barriers | 3% |
| 1 barrier | 37% |
| 2 barriers | 30% |
| 3 barriers | 9% |
| 4 barriers | 13% |
| 5 barriers | 6% |
| 6 barriers | 2% |
Barrier counts vary widely across Advertising & Marketing sites. Single-barrier configurations are the modal count at 37%, but the distribution is broad. 30% of sites carry three or more.
Most Advertising & Marketing sites are Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.48 |
A mean recommended tier of 1.48 and a median of Simple place Advertising & Marketing toward the easier end of the sectors surveyed. 63% of sites are Simple to access; 10% require Moderate or above.
Simple is the dominant tier in Advertising & Marketing, but Easy accounts for a significant share
| % of sites | |
|---|---|
| 1 - Simple | 63% |
| 2 - Easy | 27% |
| 3 - Moderate | 9% |
| 4 - Complex | 1% |
| 5 - Advanced | 0% |
Simple (63%) is the most common tier. Simple and Easy together account for 90% of sites, while 9% land at Moderate and 1% at Complex or Advanced.
robots.txt
Most Advertising & Marketing sites publish a robots.txt — coverage above the norm
| % of sites | |
|---|---|
| Publish a valid robots.txt | 87% |
87% of Advertising & Marketing sites publish a valid robots.txt. The 13% without a file leave crawl policy undefined — a lower gap than the cross-sector average of around 24%.
Advertising & Marketing sites name a handful of agents in robots.txt — few commit to comprehensive policies
| % of sites | |
|---|---|
| 0 | 30% |
| 1 | 42% |
| 2–4 | 15% |
| 5–9 | 7% |
| 10–15 | 3% |
| 16+ | 3% |
The mean of 2.9 named agents reflects limited bot-policy investment in Advertising & Marketing. 30% of sites name no agents; 42% name exactly one. Only 3% maintain a list of 16 or more named bots.
AI crawlers draw the most robots.txt attention in Advertising & Marketing
| % of sites | |
|---|---|
| AI crawlers | 10% |
| SEO / search indexing | 6% |
| SEO audit tools | 6% |
| Commercial scrapers | 2% |
| Social / content scrapers | 4% |
| Archive / research crawlers | 0% |
| Monitoring / uptime bots | 0% |
AI crawlers are named in 10% of Advertising & Marketing robots.txt files, followed by SEO / search indexing at 6%. All figures are percentages of all 100 sites including those with no robots.txt.
Uniform per-agent rules dominate Advertising & Marketing robots.txt files
| % of sites | |
|---|---|
| Uniform per-agent rules | 47% |
| Path-level mixed rules | 29% |
| Allow-only (no Disallow rules) | 7% |
| Mixed AI posture | 0% |
| Blanket Disallow: / | 4% |
Uniform per-agent rules lead at 47%, with path-level mixed rules at 29%. Mixed AI posture (0%) and blanket Disallow (4%) are minimal.
AI crawlers are among the most-targeted agents in Advertising & Marketing robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 60% | 39% |
| ClaudeBot | 6% | 2% |
| GPTBot | 6% | 1% |
| Google-Extended | 5% | 2% |
| Amazonbot | 6% | — |
| CCBot | 5% | 1% |
| Applebot-Extended | 5% | — |
| Bingbot | 4% | 1% |
| Googlebot | 4% | 1% |
| PerplexityBot | 2% | 3% |
| Twitterbot | 2% | 3% |
| AhrefsBot | 4% | 1% |
| Bytespider | 5% | — |
| meta-externalagent | 5% | — |
| Googlebot-Image | 2% | 2% |
| Pinterestbot | 3% | 1% |
| facebookexternalhit | 2% | 2% |
| OAI-SearchBot | 1% | 3% |
| AdsBot-Google | 1% | 3% |
| GoogleBot | 2% | 2% |
| Applebot | 2% | 1% |
| DuckDuckBot | 2% | 1% |
| LinkedInBot | 1% | 2% |
| Slurp | 2% | 1% |
| Yandex | 2% | 1% |
Values are percentages of all 100 Advertising & Marketing sites surveyed. The wildcard (*) leads at 60% Disallow. Among named agents, ClaudeBot, GPTBot, and Amazonbot share the top Disallow count — reflecting direct targeting of AI training and indexing bots.
Industry Cohort
Advertising & Marketing is the most accessible sub-category in the Media & Publishing cohort
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Advertising & Marketing | 97% | 23% | 32% | 31% | 21% | 14% | 87% | 6% | Tier 1 |
| Mass Media | 98% | 12% | 21% | 49% | 16% | 8% | 92% | 43% | Tier 2 |
| Newspapers | 96% | 8% | 21% | 36% | 15% | 10% | 90% | 64% | Tier 2 |
| Public Relations | 95% | 10% | 14% | 25% | 14% | 14% | 76% | 6% | Tier 1 |
| Publishing | 95% | 16% | 24% | 45% | 21% | 11% | 92% | 49% | Tier 2 |
| Writing & Editing | 98% | 14% | 33% | 31% | 11% | 6% | 79% | 11% | Tier 1 |
A median tier of Simple and mean of 1.41 make Advertising & Marketing the easiest to access programmatically within Media & Publishing. WAF (97%) and AI blocking (6%) are in line with cohort peers.