Overview
JavaScript leads Consulting's secondary barrier stack above the norm
| Value | |
|---|---|
| WAF | 87% |
| Antibot | 9% |
| CAPTCHA | 22% |
| JavaScript | 37% |
| Rate Limiting | 8% |
| TLS | 11% |
WAF at 87% is the dominant barrier. JavaScript (37%) leads secondary coverage. The remaining barriers sit materially lower, giving Consulting a narrow but defined secondary layer.
Barrier Landscape
WAF leads Consulting at 87% — JavaScript is the main secondary layer
| Value | |
|---|---|
| WAF | 87% |
| Antibot | 9% |
| CAPTCHA | 22% |
| JavaScript | 37% |
| Rate Limiting | 8% |
| TLS | 11% |
| Has robots.txt | 75% |
| Blocks AI crawlers | 5% |
WAF dominates at 87%. JavaScript (37%) leads active secondary barriers, followed by CAPTCHA (22%). The remaining three barriers all sit at or below 11% — a concentrated rather than broadly stacked profile.
Two-barrier stacks are Consulting's most common security posture
| % of sites | |
|---|---|
| 0 barriers | 10% |
| 1 barrier | 31% |
| 2 barriers | 43% |
| 3 barriers | 10% |
| 4 barriers | 3% |
| 5 barriers | 3% |
| 6 barriers | 0% |
43% of Consulting sites carry exactly 2 barriers. 16% stack three or more. 10% carry none — a moderate share of unprotected sites for this sector.
Consulting sites are predominantly Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.23 |
A mean recommended tier of 1.23 and a median of Simple place Consulting among the most accessible sectors surveyed. 81% of sites are Simple to access; 2% require Moderate or above.
Simple tier covers more than three-quarters of Consulting sites
| % of sites | |
|---|---|
| 1 - Simple | 81% |
| 2 - Easy | 17% |
| 3 - Moderate | 0% |
| 4 - Complex | 2% |
| 5 - Advanced | 0% |
Simple (81%) is the most common tier. Simple and Easy together account for 98% of sites, while 0% land at Moderate and 2% at Complex or Advanced.
robots.txt
Three-quarters of Consulting sites publish a robots.txt
| % of sites | |
|---|---|
| Publish a valid robots.txt | 75% |
75% of Consulting sites have a valid robots.txt. The remaining 25% leave crawl policy entirely undefined — a meaningful gap for a sector of this scale.
Most Consulting sites name one bot or none in their robots.txt
| % of sites | |
|---|---|
| 0 | 40% |
| 1 | 35% |
| 2–4 | 13% |
| 5–9 | 8% |
| 10–15 | 1% |
| 16+ | 3% |
40% of Consulting sites name zero agents and 35% name one. A mean of 2.1 is pulled up by a small number of sites with extensive policies, while the median stays at 1.
AI crawlers draw the most robots.txt attention in Consulting
| % of sites | |
|---|---|
| AI crawlers | 7% |
| SEO / search indexing | 6% |
| SEO audit tools | 7% |
| Commercial scrapers | 1% |
| Social / content scrapers | 2% |
| Archive / research crawlers | 3% |
| Monitoring / uptime bots | 0% |
AI crawlers are named in 7% of Consulting robots.txt files, tied with SEO audit tools at 7%. All figures are percentages of all 100 sites including those with no robots.txt.
Uniform per-agent rules dominate Consulting robots.txt files
| % of sites | |
|---|---|
| Uniform per-agent rules | 44% |
| Path-level mixed rules | 25% |
| Allow-only (no Disallow rules) | 5% |
| Mixed AI posture | 2% |
| Blanket Disallow: / | 0% |
Uniform per-agent rules lead at 44%, with path-level mixed rules at 25%. Mixed AI posture (2%) and blanket Disallow (0%) are minimal.
CCBot and MJ12bot lead named Disallow rules in Consulting robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 51% | 26% |
| CCBot | 4% | 1% |
| GPTBot | 3% | 2% |
| Google-Extended | 3% | 2% |
| MJ12bot | 4% | — |
| ClaudeBot | 2% | 2% |
| Googlebot | 2% | 1% |
| Bytespider | 2% | 1% |
| PerplexityBot | 0% | 3% |
| Anthropic-Bot | 1% | 1% |
| BlackWidow | 2% | — |
| EmailCollector | 2% | — |
| EmailSiphon | 2% | — |
| HTTP Weazel | 2% | — |
| HTTrack | 2% | — |
| HTTrack Website Copier | 2% | — |
| Leech | 2% | — |
| Offline Commander | 2% | — |
| Teleport | 2% | — |
| TeleportPro | 2% | — |
| Web Downloader | 2% | — |
| WebBandit | 2% | — |
| WebCopier | 2% | — |
| WebReaper | 2% | — |
| WebSnake | 2% | — |
Values are percentages of all 100 Consulting sites surveyed. The wildcard (*) leads at 51% Disallow. Among named agents, CCBot and MJ12bot tie at 4% Disallow — followed by GPTBot and Google-Extended at 3%.
Industry Cohort
Consulting is the most accessible sub-category in the Legal & Professional Services cohort
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Consulting | 87% | 9% | 22% | 37% | 8% | 11% | 75% | 5% | Tier 1 |
| Customer Services | 100% | 22% | 22% | 55% | 38% | 17% | 76% | 7% | Tier 1 |
| Facilities Services | 94% | 9% | 23% | 27% | 12% | 10% | 75% | 1% | Tier 1 |
| Legal | 83% | 15% | 22% | 35% | 15% | 8% | 58% | 5% | Tier 2 |
A median tier of Simple and mean of 1.22 make Consulting the easiest to access programmatically within Legal & Professional Services. WAF (87%) and AI blocking (5%) are in line with cohort peers.