Overview
JavaScript is BVI's most common secondary barrier
| Value | |
|---|---|
| WAF | 93% |
| Antibot | 22.6% |
| CAPTCHA | 23.9% |
| JavaScript | 43.4% |
| Rate Limiting | 15.3% |
| TLS | 17.2% |
WAF is near-universal at 93.0%. JavaScript at 43.4% leads secondary barriers, ahead of CAPTCHA (23.9%) and Antibot (22.6%). TLS (17.2%) and Rate Limiting (15.3%) are the lowest.
Barrier Landscape
JavaScript tops BVI's active secondary defences
| Value | |
|---|---|
| WAF | 93% |
| Antibot | 22.6% |
| CAPTCHA | 23.9% |
| JavaScript | 43.4% |
| Rate Limiting | 15.3% |
| TLS | 17.2% |
| Has robots.txt | 81.2% |
| Blocks AI crawlers | 15% |
WAF reaches 93.0% across BVI’s 725 domains. JavaScript follows at 43.4% — the only other barrier above 23%. CAPTCHA (23.9%), Antibot (22.6%), TLS (17.2%), and Rate Limiting (15.3%) fill the remaining positions.
Most BVI sites carry at least two barriers
| % of sites | |
|---|---|
| 0 barriers | 3.6% |
| 1 barrier | 29.8% |
| 2 barriers | 33.9% |
| 3 barriers | 19% |
| 4 barriers | 8.1% |
| 5 barriers | 5% |
| 6 barriers | 0.6% |
Two-barrier sites are the mode at 33.9%, with one-barrier sites at 29.8%. Only 3.6% of sites have no barriers. Sites carrying three or more barriers total 33.3% — a substantial share.
BVI sites sit comfortably in the Easy access band
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
Browser-based median recommended tier is 2, meaning fewer than half of BVI’s domains are retrievable at Tier 1. The mean of 1.69 confirms most sites cluster at Tiers 1 and 2, with few requiring higher.
Most BVI sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 49.1% |
| 2 - Easy | 37% |
| 3 - Moderate | 10.8% |
| 4 - Complex | 2.5% |
| 5 - Advanced | 0.7% |
Tier 1 covers 49.1% of sites and Tier 2 a further 37.0% — 86.1% combined. Tier 3 accounts for 10.8%. Tiers 4 and 5 together affect 3.2% of domains.
robots.txt
AI crawlers top BVI's robots.txt bot categories
| % of sites | |
|---|---|
| AI crawlers | 17% |
| SEO / search indexing | 13.1% |
| SEO audit tools | 16.3% |
| Commercial scrapers | 7.4% |
| Social / content scrapers | 6.9% |
| Archive / research crawlers | 6.1% |
| Monitoring / uptime bots | 0.6% |
AI crawlers appear in 17.0% of BVI sites’ robot policies — the top category — followed closely by SEO audit tools at 16.3% and SEO/search indexing at 13.1%. Commercial scrapers are named by 7.4% of sites.
Uniform rules are BVI's most common robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 43.2% |
| Path-level mixed rules for same agent | 33% |
| Allow-only (no disallow rules) | 1.8% |
| Mixed AI posture (block some, allow others) | 3% |
| Blanket Disallow: / | 2.9% |
Uniform per-agent rules account for 43.2% of all BVI sites and path-level mixed rules for 33.0%. Blanket Disallow: / applies to 2.9% of sites. Allow-only postures are rare at 1.8%.
GPTBot faces the most Disallow rules in BVI's robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.9% | 1.1% |
| CCBot | 8.7% | 0.7% |
| ClaudeBot | 9.8% | 0.3% |
| Google-Extended | 9.9% | 1% |
| Bytespider | 8% | 0.3% |
| meta-externalagent | 4.7% | 0.4% |
| ChatGPT-User | 6.9% | 2.3% |
| PerplexityBot | 7.4% | 1.9% |
| Applebot-Extended | 5.1% | 0.1% |
| OAI-SearchBot | 5.1% | 1.8% |
| anthropic-ai | 4.7% | 0.6% |
GPTBot is named in Disallow rules on 11.9% of all BVI sites. Google-Extended (9.9%) and ClaudeBot (9.8%) follow. ChatGPT-User (2.3%) receives the most explicit Allow rules.
Six BVI sectors all name at least one bot to restrict
| % of sites | |
|---|---|
| Agriculture | 100% |
| Design | 100% |
| Entertainment | 100% |
| Machinery | 100% |
| Medical Devices | 100% |
| Retail | 100% |
| Advertising & Mktg | 90% |
| Beauty & Cosmetics | 90% |
| Computer & Video Games | 90% |
| Events | 90% |
| General | 90% |
| Mass Media | 90% |
| Newspapers | 90% |
| Online Services | 90% |
| Photography | 90% |
Agriculture, Design, Entertainment, Machinery, Medical Devices, and Retail all reach 100% — every site names at least one bot in a Disallow rule. Nine further sectors sit at 90%, including Advertising & Marketing and Newspapers.
BVI's Market Research and Photography sectors lead in explicitly allowing crawlers
| % of sites | |
|---|---|
| Market Research | 80% |
| Photography | 80% |
| Machinery | 71.4% |
| Publishing | 70% |
| Retail | 70% |
| Airlines | 60% |
| Beauty & Cosmetics | 60% |
| Healthcare | 60% |
| Medical Devices | 60% |
| Real Estate | 60% |
| Restaurants | 60% |
| Software & Dev | 60% |
| Sports | 60% |
| Telecom | 60% |
| Travel & Tourism | 60% |
Market Research and Photography each reach 80% — eight in ten sites name at least one bot in an Allow rule. Machinery (71.4%) and Publishing and Retail (70.0% each) follow closely.
Industry Patterns
Computer Hardware is BVI's most demanding sector for crawlers
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Computer Hardware | 80% | 10% | 0% | 30% | 20% | 30% | 70% | 0% | Tier 3 |
| Jewellery & Luxury | 80% | 30% | 20% | 50% | 10% | 30% | 70% | 10% | Tier 3 |
| Apparel & Fashion | 90% | 10% | 20% | 70% | 20% | 20% | 80% | 0% | Tier 2 |
| Retail | 100% | 10% | 0% | 70% | 10% | 10% | 100% | 40% | Tier 2 |
| Sporting Goods | 80% | 20% | 30% | 40% | 30% | 30% | 60% | 10% | Tier 2 |
| Beauty & Cosmetics | 100% | 50% | 50% | 50% | 60% | 50% | 90% | 10% | Tier 2 |
| Real Estate | 100% | 20% | 10% | 50% | 30% | 20% | 80% | 20% | Tier 2 |
| Equipment & Supplies | 100% | 12.5% | 37.5% | 50% | 25% | 0% | 75% | 12.5% | Tier 2 |
| Food & Beverages | 100% | 60% | 30% | 80% | 40% | 10% | 80% | 10% | Tier 2 |
| Consumer Electronics | 80% | 40% | 30% | 50% | 20% | 50% | 90% | 30% | Tier 2 |
| Hospitality | 80% | 30% | 20% | 50% | 10% | 40% | 60% | 10% | Tier 2 |
| Human Resources | 100% | 40% | 10% | 30% | 30% | 10% | 80% | 0% | Tier 2 |
| Mail & Delivery | 70% | 30% | 20% | 10% | 10% | 20% | 60% | 0% | Tier 2 |
| Market Research | 80% | 0% | 20% | 80% | 0% | 0% | 80% | 0% | Tier 2 |
| Transport & Logistics | 70% | 30% | 20% | 20% | 10% | 20% | 60% | 0% | Tier 2 |
Computer Hardware leads at median Moderate. Jewellery & Luxury also reaches median Moderate. Apparel & Fashion, Retail, and Sporting Goods sit at the same mean tier, despite returning a median of Easy. All other top-15 sectors cluster at median Easy.
Multiple BVI sectors share the most accessible floor tier
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Wholesalers | 90% | 20% | 40% | 50% | 30% | 10% | 90% | 20% | Tier 2 |
| Design | 100% | 20% | 30% | 50% | 20% | 20% | 100% | 30% | Tier 2 |
| Restaurants | 90% | 30% | 40% | 50% | 20% | 10% | 90% | 10% | Tier 2 |
| Travel & Tourism | 100% | 40% | 10% | 60% | 30% | 20% | 80% | 30% | Tier 2 |
| Manufacturing | 83.3% | 16.7% | 0% | 50% | 0% | 16.7% | 83.3% | 16.7% | Tier 2 |
| Airlines | 90% | 10% | 10% | 50% | 30% | 30% | 90% | 30% | Tier 2 |
| Automotive | 100% | 10% | 0% | 50% | 50% | 10% | 80% | 20% | Tier 2 |
| Education | 100% | 20% | 10% | 40% | 20% | 10% | 80% | 10% | Tier 2 |
| Finance | 80% | 20% | 0% | 10% | 10% | 10% | 70% | 30% | Tier 2 |
| Healthcare | 100% | 20% | 20% | 30% | 10% | 10% | 90% | 20% | Tier 2 |
| Legal | 100% | 0% | 10% | 40% | 0% | 10% | 80% | 10% | Tier 2 |
| Music | 100% | 20% | 20% | 60% | 40% | 0% | 70% | 20% | Tier 2 |
| Non-Profit | 90% | 20% | 10% | 70% | 10% | 0% | 70% | 10% | Tier 2 |
| Publishing | 90% | 30% | 30% | 50% | 20% | 0% | 90% | 60% | Tier 2 |
| Science | 80% | 30% | 10% | 30% | 20% | 20% | 70% | 10% | Tier 2 |
The most accessible sectors in BVI — including Airlines, Automotive, Education, Finance, Healthcare, Legal, Music, Non-Profit, Publishing, and Science — all share a mean recommended tier of 1.80.