Overview
JavaScript leads Palau's secondary barriers, with CAPTCHA a distant second
| Value | |
|---|---|
| WAF | 93.9% |
| Antibot | 18.6% |
| CAPTCHA | 27.1% |
| JavaScript | 40.7% |
| Rate Limiting | 15.7% |
| TLS | 14.8% |
WAF dominates at 93.9%. JavaScript is the top secondary barrier at 40.7%, followed by CAPTCHA at 27.1% and Antibot at 18.6%. Rate Limiting (15.7%) and TLS (14.8%) trail at the lower end.
Barrier Landscape
JavaScript outranks Antibot in Palau's secondary barriers
| Value | |
|---|---|
| WAF | 93.9% |
| Antibot | 18.6% |
| CAPTCHA | 27.1% |
| JavaScript | 40.7% |
| Rate Limiting | 15.7% |
| TLS | 14.8% |
| Has robots.txt | 81.4% |
| Blocks AI crawlers | 22% |
WAF leads at 93.9%. JavaScript follows at 40.7%. CAPTCHA reaches 27.1% against Antibot's 18.6%. Rate Limiting (15.7%) exceeds TLS (14.8%) at the lower end of the barrier spectrum.
Two-barrier sites are the most common configuration
| % of sites | |
|---|---|
| 0 barriers | 3.1% |
| 1 barrier | 31.7% |
| 2 barriers | 34.6% |
| 3 barriers | 19.1% |
| 4 barriers | 4.8% |
| 5 barriers | 6.1% |
| 6 barriers | 0.5% |
3.1% of sites carry no barrier. The modal count is two barriers at 34.6%, followed closely by one barrier at 31.7%. Sites with three or more barriers total 30.5%, and 0.5% deploy all six.
Most Palau sites are Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.6 |
Median recommended tier is 1: more than half of sites are retrievable with simple HTTP. Mean of 1.64 reflects upward pressure from the 11.6% at Tier 3 — the highest Tier 3 share among this group.
Moderate-difficulty sites reach an elevated share in Palau
| % of sites | |
|---|---|
| 1 - Simple | 54% |
| 2 - Easy | 31.7% |
| 3 - Moderate | 11.6% |
| 4 - Complex | 1.7% |
| 5 - Advanced | 1% |
Tier 1 accounts for 54.0% and Tier 2 for 31.7%, totalling 85.7%. Tier 3 is notable at 11.6%. Tiers 4 and 5 remain marginal at 1.7% and 1.0% respectively.
robots.txt
AI crawlers are named by nearly a quarter of all Palau sites
| % of sites | |
|---|---|
| AI crawlers | 23.2% |
| SEO / search indexing | 13.3% |
| SEO audit tools | 15.3% |
| Commercial scrapers | 10.7% |
| Social / content scrapers | 8% |
| Archive / research crawlers | 4.8% |
| Monitoring / uptime bots | 0.2% |
AI crawlers appear in 23.2% of all sites — the highest category. SEO audit tools (15.3%) again outrank SEO / search indexing (13.3%). Commercial scrapers are mentioned by 10.7% of sites.
Uniform rules are the most common posture in Palau
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 45.5% |
| Path-level mixed rules for same agent | 30.3% |
| Allow-only (no disallow rules) | 1.7% |
| Mixed AI posture (block some, allow others) | 2.9% |
| Blanket Disallow: / | 3.4% |
Uniform per-agent rules apply at 45.5% of sites. Path-level mixed rules account for 30.3%. Blanket Disallow: / reaches 3.4%, slightly above Mixed AI posture at 2.9%. Allow-only is the least common at 1.7%.
GPTBot is the most blocked agent in Palau
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 17.4% | 0.5% |
| CCBot | 14.8% | 0.7% |
| ClaudeBot | 14.3% | 0.2% |
| Google-Extended | 11.4% | 0.2% |
| Bytespider | 10.9% | 0.7% |
| meta-externalagent | 4.4% | 0.2% |
| ChatGPT-User | 10.7% | 2.2% |
| PerplexityBot | 10.7% | 2.2% |
| Applebot-Extended | 7.7% | 0.2% |
| OAI-SearchBot | 5.6% | 1.9% |
| anthropic-ai | 9.9% | 0.2% |
GPTBot is named in Disallow rules on 17.4% of all sites, the largest single-agent block rate. CCBot follows at 14.8% and ClaudeBot at 14.3%. ChatGPT-User has the highest Allow rate at 2.2%.
Five sectors — including Maritime — block every named bot
| % of sites | |
|---|---|
| Design | 100% |
| Entertainment | 100% |
| Maritime | 100% |
| Newspapers | 100% |
| Publishing | 100% |
| Airlines | 90% |
| Apparel & Fashion | 90% |
| Distance Learning | 90% |
| General | 90% |
| Mass Media | 90% |
| Online Services | 90% |
| Retail | 90% |
| Computer & Video Games | 80% |
| Consumer Electronics | 80% |
| Equipment & Supplies | 80% |
Design, Entertainment, Maritime, Newspapers, and Publishing each have every sampled site naming at least one bot in a Disallow rule. Six further sectors sit at 80–90%.
Palau's Software & Dev sites lead in explicitly allowing crawlers
| % of sites | |
|---|---|
| Software & Dev | 80% |
| Airlines | 70% |
| Sports | 70% |
| Ecology & Environment | 66.7% |
| Photography | 62.5% |
| Consumer Electronics | 60% |
| IT Services | 60% |
| Publishing | 60% |
| Retail | 60% |
| Travel & Tourism | 60% |
| Adult | 50% |
| Computer Hardware | 50% |
| General | 50% |
| Maritime | 50% |
| Mass Media | 50% |
Software & Dev tops the explicit bot Allow ranking at 80% of sites naming at least one bot to permit. Airlines and Sports both reach 70%. Ecology & Environment follows at 66.7%.
Industry Patterns
Retail is the hardest sector to access programmatically in Palau
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Retail | 100% | 30% | 10% | 50% | 10% | 0% | 90% | 20% | Tier 2 |
| Automotive | 100% | 20% | 10% | 40% | 10% | 0% | 80% | 10% | Tier 2 |
| Non-Profit | 90% | 10% | 20% | 30% | 10% | 20% | 70% | 30% | Tier 2 |
| Adult | 100% | 40% | 40% | 60% | 10% | 20% | 90% | 20% | Tier 2 |
| Apparel & Fashion | 90% | 20% | 30% | 60% | 10% | 30% | 90% | 10% | Tier 2 |
| Travel & Tourism | 90% | 30% | 10% | 70% | 10% | 30% | 80% | 20% | Tier 2 |
| Hospitality | 100% | 20% | 30% | 70% | 20% | 10% | 50% | 0% | Tier 2 |
| Music | 100% | 30% | 30% | 40% | 20% | 0% | 100% | 30% | Tier 2 |
| Airlines | 80% | 30% | 10% | 70% | 10% | 30% | 90% | 20% | Tier 2 |
| Publishing | 100% | 20% | 40% | 30% | 20% | 0% | 100% | 50% | Tier 2 |
| Government | 80% | 30% | 20% | 60% | 30% | 20% | 70% | 10% | Tier 2 |
| Healthcare | 100% | 20% | 50% | 30% | 20% | 10% | 90% | 60% | Tier 2 |
| Science | 88.9% | 22.2% | 0% | 22.2% | 22.2% | 11.1% | 55.6% | 0% | Tier 2 |
| Consumer Electronics | 100% | 20% | 20% | 20% | 20% | 40% | 80% | 0% | Tier 2 |
| Distance Learning | 100% | 20% | 50% | 20% | 10% | 10% | 90% | 40% | Tier 2 |
The hardest sector is Retail, with a median recommended tier of 2 and mean of 2.50. Automotive (mean 2.35) and Non-Profit (2.30) follow. All 15 sectors carry a median of 2, with means ranging 1.60–2.50.
Seven sectors share the most accessible floor in Palau
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Education | 100% | 20% | 20% | 50% | 20% | 0% | 80% | 10% | Tier 2 |
| Computer Hardware | 83.3% | 0% | 16.7% | 16.7% | 0% | 16.7% | 66.7% | 0% | Tier 2 |
| Ecology & Environment | 100% | 50% | 16.7% | 16.7% | 0% | 0% | 100% | 0% | Tier 2 |
| Real Estate | 80% | 40% | 40% | 60% | 40% | 20% | 80% | 0% | Tier 1 |
| Maritime | 100% | 33.3% | 33.3% | 33.3% | 16.7% | 16.7% | 100% | 33.3% | Tier 1 |
| Design | 100% | 12.5% | 25% | 25% | 12.5% | 12.5% | 100% | 37.5% | Tier 1 |
| Finance | 90% | 20% | 0% | 20% | 30% | 10% | 70% | 30% | Tier 1 |
| General | 100% | 0% | 50% | 50% | 10% | 40% | 90% | 40% | Tier 1 |
| Entertainment | 90% | 10% | 60% | 50% | 10% | 20% | 100% | 60% | Tier 1 |
| Mass Media | 100% | 0% | 50% | 50% | 10% | 40% | 90% | 40% | Tier 1 |
| Medical Devices | 100% | 16.7% | 33.3% | 66.7% | 16.7% | 0% | 50% | 0% | Tier 1 |
| Newspapers | 100% | 0% | 30% | 30% | 10% | 10% | 100% | 80% | Tier 1 |
| Online Services | 100% | 0% | 50% | 50% | 10% | 40% | 90% | 40% | Tier 1 |
| Photography | 87.5% | 25% | 25% | 50% | 25% | 37.5% | 75% | 12.5% | Tier 1 |
| Wellness | 100% | 30% | 60% | 50% | 50% | 10% | 90% | 60% | Tier 1 |
The floor mean of 1.50 is shared by ten sectors — all with median tier 1. Finance (mean 1.60) and Design (1.62) sit just above this floor.