Overview
JavaScript and CAPTCHA both elevate Agriculture's active barrier profile
| Value | |
|---|---|
| WAF | 97% |
| Antibot | 12% |
| CAPTCHA | 30% |
| JavaScript | 41% |
| Rate Limiting | 20% |
| TLS | 12% |
WAF leads at 97%. JavaScript (41%) and CAPTCHA (30%) together form a multi-layer active stack — more than most sectors deploy alongside WAF.
Barrier Landscape
WAF leads Agriculture at 97% — JavaScript is the main secondary layer
| Value | |
|---|---|
| WAF | 97% |
| Antibot | 12% |
| CAPTCHA | 30% |
| JavaScript | 41% |
| Rate Limiting | 20% |
| TLS | 12% |
| Has robots.txt | 90% |
| Blocks AI crawlers | 12% |
WAF dominates at 97%. JavaScript (41%) leads active secondary barriers, followed by CAPTCHA (30%). The remaining three barriers all sit at or below 20% — a concentrated rather than broadly stacked profile.
Two-barrier stacks are Agriculture's most common security posture
| % of sites | |
|---|---|
| 0 barriers | 2% |
| 1 barrier | 30% |
| 2 barriers | 40% |
| 3 barriers | 18% |
| 4 barriers | 3% |
| 5 barriers | 6% |
| 6 barriers | 1% |
40% of Agriculture sites carry exactly 2 barriers. 28% stack three or more. 2% carry none — a low share of unprotected sites for this sector.
Agriculture sites are predominantly Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.34 |
A mean recommended tier of 1.34 and a median of Simple place Agriculture toward the easier end of the sectors surveyed. 76% of sites are Simple to access; 7% require Moderate or above.
Simple tier covers more than three-quarters of Agriculture sites
| % of sites | |
|---|---|
| 1 - Simple | 76% |
| 2 - Easy | 17% |
| 3 - Moderate | 4% |
| 4 - Complex | 3% |
| 5 - Advanced | 0% |
Simple (76%) is the most common tier. Simple and Easy together account for 93% of sites, while 4% land at Moderate and 3% at Complex or Advanced.
robots.txt
Most Agriculture sites publish a robots.txt — coverage above the norm
| % of sites | |
|---|---|
| Publish a valid robots.txt | 90% |
90% of Agriculture sites publish a valid robots.txt. The 10% without a file leave crawl policy undefined — a lower gap than the cross-sector average of around 24%.
A large minority of Agriculture sites maintain exhaustive bot agent lists
| % of sites | |
|---|---|
| 0 | 14% |
| 1 | 47% |
| 2–4 | 10% |
| 5–9 | 16% |
| 10–15 | 2% |
| 16+ | 11% |
A mean of 20 named agents is driven by a cohort of sites with comprehensive robots.txt policies. 11% name 16 or more agents. The median stays at 1, meaning most sites are sparse but the outliers are extensive.
SEO audit tools lead Agriculture's robots.txt mentions ahead of AI crawlers
| % of sites | |
|---|---|
| AI crawlers | 14% |
| SEO / search indexing | 14% |
| SEO audit tools | 24% |
| Commercial scrapers | 9% |
| Social / content scrapers | 3% |
| Archive / research crawlers | 6% |
| Monitoring / uptime bots | 1% |
SEO audit tools are mentioned by 24% of Agriculture sites — the leading bot category, ahead of AI crawlers at 14%. Figures are percentages of all 100 sites surveyed.
Uniform per-agent rules dominate Agriculture robots.txt files
| % of sites | |
|---|---|
| Uniform per-agent rules | 57% |
| Path-level mixed rules | 29% |
| Allow-only (no Disallow rules) | 4% |
| Mixed AI posture | 0% |
| Blanket Disallow: / | 0% |
Uniform per-agent rules lead at 57%, with path-level mixed rules at 29%. Mixed AI posture (0%) and blanket Disallow (0%) are minimal.
AhrefsBot leads named Disallow rules in Agriculture robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 77% | 34% |
| AhrefsBot | 19% | 2% |
| Nutch | 12% | — |
| AhrefsSiteAudit | 11% | 1% |
| CCBot | 11% | — |
| GPTBot | 10% | 1% |
| Amazonbot | 10% | — |
| Google-Extended | 9% | 1% |
| MJ12bot | 10% | — |
| adsbot-google | 10% | — |
| ClaudeBot | 8% | 1% |
| Bytespider | 8% | — |
| BLEXBot | 7% | — |
| Baiduspider | 7% | — |
| ChatGPT-User | 6% | 1% |
| SemrushBot | 7% | — |
| TurnitinBot | 7% | — |
| dotbot | 7% | — |
| Applebot-Extended | 6% | — |
| Googlebot | 4% | 2% |
| Claude-Web | 4% | 1% |
| Diffbot | 4% | 1% |
| PerplexityBot | 4% | 1% |
| anthropic-ai | 4% | 1% |
| cohere-ai | 4% | 1% |
Values are percentages of all 100 Agriculture sites surveyed. The wildcard (*) leads at 77% Disallow. Among named agents, AhrefsBot, Nutch, AhrefsSiteAudit are the most frequently targeted — focused on non-AI crawlers.
Industry Cohort
Agriculture leads the Agriculture cohort on AI-blocking policy
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Agriculture | 97% | 12% | 30% | 41% | 20% | 12% | 90% | 12% | Tier 1 |
| Farms & Ranches | 96% | 12% | 32% | 26% | 24% | 14% | 67% | 6% | Tier 1 |
| Fishery | 96% | 13% | 37% | 32% | 18% | 9% | 69% | 3% | Tier 1 |
Agriculture's 12% AI-blocking rate in robots.txt is the highest in the Agriculture cohort. WAF (97%) and JavaScript (41%) sit at the cohort on technical barriers.