Overview
JavaScript leads secondary barriers across Norfolk Island sites
| Value | |
|---|---|
| WAF | 93.7% |
| Antibot | 18% |
| CAPTCHA | 26.4% |
| JavaScript | 37.4% |
| Rate Limiting | 10.3% |
| TLS | 17% |
WAF covers 93.7% of sites. Among secondary barriers, JavaScript reaches 37.4% — well ahead of CAPTCHA at 26.4% and Antibot at 18.0%. TLS (17.0%) and Rate Limiting (10.3%) are the least deployed.
Barrier Landscape
JavaScript leads secondary barriers in Norfolk Island
| Value | |
|---|---|
| WAF | 93.7% |
| Antibot | 18% |
| CAPTCHA | 26.4% |
| JavaScript | 37.4% |
| Rate Limiting | 10.3% |
| TLS | 17% |
| Has robots.txt | 82% |
| Blocks AI crawlers | 13.6% |
WAF dominates at 93.7%. JavaScript is the next most common barrier at 37.4%, followed by CAPTCHA at 26.4% and Antibot at 18.0%. TLS (17.0%) and Rate Limiting (10.3%) are the least deployed.
Single-barrier sites are the most common outcome
| % of sites | |
|---|---|
| 0 barriers | 4.3% |
| 1 barrier | 33.5% |
| 2 barriers | 32.9% |
| 3 barriers | 17.9% |
| 4 barriers | 7.7% |
| 5 barriers | 3.2% |
| 6 barriers | 0.5% |
33.5% of sites carry exactly one barrier and 32.9% carry two — the two categories nearly tied. Zero-barrier sites account for 4.3%. Sites with three or more barriers make up 29.3% of the total.
Most Norfolk Island sites are Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.5 |
The median recommended tier is 1 and the mean is 1.54, indicating that the majority of Norfolk Island sites are retrievable at the simplest access level. 57.6% sit at Tier 1.
Over nine in 10 sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 57.6% |
| 2 - Easy | 33.9% |
| 3 - Moderate | 6.4% |
| 4 - Complex | 1.6% |
| 5 - Advanced | 0.5% |
Tier 1 accounts for 57.6% of sites and Tier 2 for 33.9%, giving a combined 91.5%. Tier 3 adds 6.4%, while Tiers 4 and 5 together reach just 2.1%.
robots.txt
SEO audit tools edge out AI crawlers in robots.txt mentions
| % of sites | |
|---|---|
| AI crawlers | 15.1% |
| SEO / search indexing | 11.7% |
| SEO audit tools | 15.7% |
| Commercial scrapers | 7.5% |
| Social / content scrapers | 7% |
| Archive / research crawlers | 4% |
| Monitoring / uptime bots | 0.5% |
SEO audit tools are named in 15.7% of all Norfolk Island sites — marginally ahead of AI crawlers at 15.1%. SEO/search indexing follows at 11.7%. Commercial scrapers (7.5%) and social bots (7.0%) are less commonly addressed.
Uniform rules are the most common posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 41.7% |
| Path-level mixed rules for same agent | 34.7% |
| Allow-only (no disallow rules) | 2.4% |
| Mixed AI posture (block some, allow others) | 3.2% |
| Blanket Disallow: / | 3% |
41.7% of sites apply uniform per-agent rules with no path mixing. Path-level mixed rules account for 34.7%. Mixed AI posture — blocking some agents while allowing others — applies to 3.2% of sites. Blanket Disallow: / covers 3.0%.
GPTBot faces the most Disallow rules in Norfolk Island's robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 10.3% | 2.7% |
| CCBot | 8.1% | 1.2% |
| ClaudeBot | 8.4% | 1.5% |
| Google-Extended | 8% | 2.3% |
| Bytespider | 7.4% | 0.7% |
| meta-externalagent | 3.1% | 0.2% |
| ChatGPT-User | 5.5% | 3.1% |
| PerplexityBot | 5.9% | 2.6% |
| Applebot-Extended | 4.5% | 0.5% |
| OAI-SearchBot | 3.8% | 2.5% |
| anthropic-ai | 4.7% | 1.1% |
GPTBot carries a Disallow rule on 10.3% of all sites, the highest block rate of any agent. ClaudeBot (8.4%) and CCBot (8.1%) follow. ChatGPT-User leads Allow mentions at 3.1%.
Five sectors unanimously name at least one bot to restrict
| % of sites | |
|---|---|
| Design | 100% |
| Furniture | 100% |
| Jewellery & Luxury | 100% |
| Retail | 100% |
| Telecom | 100% |
| Adult | 90% |
| Apparel & Fashion | 90% |
| Beauty & Cosmetics | 90% |
| Computer & Video Games | 90% |
| Consumer Electronics | 90% |
| Entertainment | 90% |
| Foreign Language | 90% |
| General | 90% |
| Mass Media | 90% |
| Newspapers | 90% |
Design, Furniture, Jewellery & Luxury, Retail, and Telecom all reach 100% — every site in each sector names at least one bot in a Disallow rule. A further ten sectors sit at 90%, indicating broad named-bot restriction across Norfolk Island industries.
Five sectors tie at the top explicit bot Allow rate
| % of sites | |
|---|---|
| Airlines | 80% |
| Automotive | 80% |
| Outsourcing | 80% |
| Publishing | 80% |
| Travel & Tourism | 80% |
| Ecology & Environment | 70% |
| Market Research | 70% |
| Photography | 70% |
| Real Estate | 70% |
| Retail | 70% |
| Telecom | 70% |
| Training & Cert. | 70% |
| Apparel & Fashion | 60% |
| Insurance | 60% |
| Plastics & Polymers | 60% |
Airlines, Automotive, Outsourcing, Publishing, and Travel & Tourism each have 80% of their sites naming at least one bot in an Allow rule — the highest rate in this ranking. Retail and Telecom also appear, each at 70%.
Industry Patterns
Real Estate and Retail share the hardest access tier
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Real Estate | 100% | 40% | 20% | 50% | 50% | 20% | 70% | 30% | Tier 2 |
| Retail | 100% | 10% | 0% | 70% | 10% | 10% | 100% | 50% | Tier 2 |
| Food & Beverages | 90% | 30% | 40% | 60% | 10% | 0% | 60% | 30% | Tier 2 |
| Human Resources | 100% | 60% | 50% | 40% | 50% | 30% | 80% | 10% | Tier 2 |
| Apparel & Fashion | 100% | 0% | 40% | 50% | 10% | 10% | 90% | 10% | Tier 2 |
| Computer Hardware | 90% | 0% | 0% | 40% | 10% | 20% | 80% | 0% | Tier 2 |
| Adult | 100% | 40% | 30% | 70% | 20% | 20% | 90% | 20% | Tier 2 |
| Consumer Electronics | 90% | 0% | 20% | 50% | 10% | 30% | 90% | 20% | Tier 2 |
| Transport & Logistics | 80% | 30% | 20% | 20% | 0% | 20% | 70% | 0% | Tier 2 |
| Travel & Tourism | 100% | 30% | 10% | 70% | 20% | 20% | 80% | 30% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Insurance | 80% | 10% | 20% | 40% | 0% | 20% | 80% | 0% | Tier 2 |
| Automotive | 90% | 20% | 0% | 30% | 20% | 40% | 80% | 40% | Tier 2 |
| Beauty & Cosmetics | 100% | 20% | 70% | 60% | 30% | 0% | 90% | 10% | Tier 2 |
| Biotech | 100% | 40% | 30% | 40% | 10% | 50% | 90% | 10% | Tier 2 |
Every sector in the top 15 sits at a median recommended tier of 2. Real Estate and Retail have the highest mean tiers at 2.50, followed by Food & Beverages and Human Resources at 2.30.
Machinery, Airlines, and Construction share the most accessible tier
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Hospitality | 90% | 20% | 0% | 60% | 0% | 20% | 60% | 10% | Tier 2 |
| Mail & Delivery | 70% | 30% | 30% | 10% | 10% | 10% | 70% | 0% | Tier 2 |
| Manufacturing | 80% | 10% | 30% | 40% | 20% | 40% | 80% | 0% | Tier 2 |
| Recruitment & Staffing | 90% | 30% | 30% | 20% | 10% | 30% | 80% | 30% | Tier 2 |
| Reference | 80% | 50% | 20% | 40% | 10% | 10% | 60% | 30% | Tier 2 |
| Science | 100% | 30% | 20% | 40% | 30% | 10% | 80% | 10% | Tier 2 |
| Telecom | 90% | 10% | 10% | 60% | 0% | 30% | 100% | 20% | Tier 2 |
| Banking | 100% | 40% | 10% | 20% | 10% | 10% | 70% | 10% | Tier 2 |
| Chemicals | 90% | 20% | 0% | 20% | 0% | 20% | 60% | 0% | Tier 2 |
| Music | 100% | 30% | 40% | 40% | 10% | 20% | 90% | 30% | Tier 2 |
| Non-Profit | 90% | 10% | 10% | 60% | 10% | 10% | 60% | 0% | Tier 2 |
| Oil & Gas | 90% | 30% | 20% | 40% | 10% | 10% | 80% | 10% | Tier 2 |
| Airlines | 100% | 10% | 10% | 60% | 20% | 20% | 90% | 10% | Tier 2 |
| Construction | 100% | 30% | 50% | 40% | 40% | 10% | 80% | 10% | Tier 2 |
| Machinery | 60% | 0% | 10% | 30% | 10% | 10% | 60% | 10% | Tier 2 |
The lowest mean recommended tier in the bottom 15 is 1.70, reached by Machinery, Airlines, and Construction. All sectors in this group have a median tier of 2, but mean scores range down to 1.70 — a higher share of Tier 1 sites.