Overview
JavaScript is Palestine's most common secondary barrier
| Value | |
|---|---|
| WAF | 92.5% |
| Antibot | 21.1% |
| CAPTCHA | 23.2% |
| JavaScript | 42.9% |
| Rate Limiting | 13.6% |
| TLS | 18.8% |
WAF is deployed by 92.5% of Palestine sites. JavaScript leads secondary barriers at 42.9%. CAPTCHA (23.2%) exceeds Antibot (21.1%) — a reversal of the typical order. TLS stands at 18.8% and Rate Limiting at 13.6%.
Barrier Landscape
CAPTCHA runs ahead of Antibot in the secondary barrier stack
| Value | |
|---|---|
| WAF | 92.5% |
| Antibot | 21.1% |
| CAPTCHA | 23.2% |
| JavaScript | 42.9% |
| Rate Limiting | 13.6% |
| TLS | 18.8% |
| Has robots.txt | 78.6% |
| Blocks AI crawlers | 14.1% |
WAF dominates at 92.5%. JavaScript leads secondary barriers at 42.9%. CAPTCHA at 23.2% is the second-ranked secondary, edging Antibot (21.1%) by more than two percentage points. Rate Limiting is lowest at 13.6%.
Two barriers is the modal site configuration
| % of sites | |
|---|---|
| 0 barriers | 5% |
| 1 barrier | 28.7% |
| 2 barriers | 34.3% |
| 3 barriers | 18% |
| 4 barriers | 9.4% |
| 5 barriers | 4.3% |
| 6 barriers | 0.3% |
34.3% of Palestine sites carry exactly two barriers — the most frequent count. Sites with three or more barriers make up 32.0% combined. A small 5.0% have no barriers. Three-barrier sites account for the largest share of the multi-barrier group at 18.0%.
Most Palestine sites are Easy to access
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
The median recommended tier is 2 and the mean is 1.70. Tier 3 accounts for 11.1% of sites — the highest among the five countries in this group. Tier 1 and Tier 2 together cover 86.2% of sites.
Moderate sites reach their highest share in this group
| % of sites | |
|---|---|
| 1 - Simple | 47.1% |
| 2 - Easy | 39.1% |
| 3 - Moderate | 11.1% |
| 4 - Complex | 2% |
| 5 - Advanced | 0.7% |
47.1% of sites are Tier 1 (Simple) and 39.1% are Tier 2 (Easy). Tier 3 (Moderate) covers 11.1% of sites. Tier 4 accounts for 2.0% and Tier 5 for 0.7%, with fewer than 3% of sites requiring advanced extraction.
robots.txt
AI crawlers lead robots.txt mentions ahead of all other categories
| % of sites | |
|---|---|
| AI crawlers | 16% |
| SEO / search indexing | 12.3% |
| SEO audit tools | 12.8% |
| Commercial scrapers | 7.6% |
| Social / content scrapers | 6.7% |
| Archive / research crawlers | 4.7% |
| Monitoring / uptime bots | 0.6% |
16.0% of all 1,071 Palestine domains name an AI crawler in their robots.txt — the highest rate of any bot category. SEO audit tools (12.8%) and SEO/search indexing bots (12.3%) follow closely. Monitoring bots are named by just 0.6%.
Uniform rules are Palestine's most common robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 40.9% |
| Path-level mixed rules for same agent | 33% |
| Allow-only (no disallow rules) | 2.1% |
| Mixed AI posture (block some, allow others) | 2.6% |
| Blanket Disallow: / | 2.4% |
40.9% of all Palestine sites use uniform per-agent rules with no path mixing. Path-level mixed rules apply to 33.0%. 21.4% publish no robots.txt. Blanket Disallow: / covers 2.4% and allow-only posture just 2.1%.
GPTBot faces the most Disallow rules in Palestine's robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.4% | 2.3% |
| CCBot | 8.5% | 0.9% |
| ClaudeBot | 8.4% | 1.1% |
| Google-Extended | 8.4% | 1.6% |
| Bytespider | 7.3% | 0.4% |
| meta-externalagent | 3.3% | 0.2% |
| ChatGPT-User | 5.7% | 2.6% |
| PerplexityBot | 5.2% | 2.1% |
| Applebot-Extended | 5% | 0.5% |
| OAI-SearchBot | 4.1% | 2.2% |
| anthropic-ai | 4.7% | 0.7% |
11.4% of all Palestine sites name GPTBot in a Disallow rule — the highest rate. CCBot follows at 8.5%. Notably, ClaudeBot and Google-Extended are tied at 8.4%. ChatGPT-User leads explicit allows at 2.6%.
Six sectors have every site naming at least one bot to restrict
| % of sites | |
|---|---|
| Apparel & Fashion | 100% |
| Consumer Electronics | 100% |
| Design | 100% |
| Entertainment | 100% |
| Newspapers | 100% |
| Retail | 100% |
| Adult | 90% |
| Agriculture | 90% |
| Beauty & Cosmetics | 90% |
| Comics & Animation | 90% |
| Computer & Video Games | 90% |
| General | 90% |
| Mass Media | 90% |
| Medical Devices | 90% |
| Online Services | 90% |
Apparel & Fashion, Consumer Electronics, Design, Entertainment, Newspapers, and Retail all reach 100% — every site in each sector names at least one bot in a Disallow rule. Nine further sectors, including Agriculture and Beauty & Cosmetics, reach 90%.
Palestine's Retail sites lead in explicitly allowing crawlers
| % of sites | |
|---|---|
| Retail | 80% |
| Airlines | 70% |
| Photography | 70% |
| Publishing | 70% |
| Science | 70% |
| Textiles & Nonwovens | 70% |
| Apparel & Fashion | 60% |
| Banking | 60% |
| Graphic Design | 60% |
| Non-Profit | 60% |
| Software & Dev | 60% |
| Sporting Goods | 60% |
| Sports | 60% |
| Travel & Tourism | 60% |
| Veterinary | 60% |
80% of Retail sites name at least one bot in an explicit Allow rule. Airlines, Photography, Publishing, Science, and Textiles & Nonwovens each reach 70%. Nine further sectors — including Non-Profit and Travel & Tourism — reach 60%.
Industry Patterns
Jewellery & Luxury and Real Estate lead the hardest-to-access sectors
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Jewellery & Luxury | 100% | 20% | 10% | 30% | 30% | 50% | 60% | 10% | Tier 3 |
| Real Estate | 100% | 10% | 10% | 60% | 40% | 30% | 50% | 20% | Tier 3 |
| Hospitality | 90% | 30% | 0% | 60% | 30% | 40% | 60% | 10% | Tier 3 |
| Human Resources | 100% | 70% | 60% | 50% | 50% | 30% | 70% | 10% | Tier 3 |
| Furniture | 90% | 30% | 40% | 70% | 30% | 30% | 70% | 0% | Tier 3 |
| Retail | 100% | 10% | 0% | 70% | 10% | 10% | 100% | 50% | Tier 2 |
| Apparel & Fashion | 90% | 0% | 20% | 50% | 20% | 40% | 100% | 10% | Tier 2 |
| Automotive | 90% | 20% | 10% | 40% | 40% | 40% | 80% | 20% | Tier 2 |
| Beauty & Cosmetics | 90% | 40% | 20% | 40% | 30% | 20% | 90% | 20% | Tier 2 |
| Customer Services | 100% | 10% | 20% | 50% | 30% | 30% | 50% | 10% | Tier 2 |
| Equipment & Supplies | 90% | 50% | 20% | 60% | 30% | 10% | 60% | 10% | Tier 2 |
| Medical Devices | 100% | 20% | 10% | 60% | 20% | 0% | 100% | 10% | Tier 2 |
| Recruitment & Staffing | 90% | 50% | 60% | 20% | 10% | 20% | 60% | 20% | Tier 2 |
| Sporting Goods | 100% | 20% | 30% | 40% | 40% | 20% | 80% | 0% | Tier 2 |
| Computer Hardware | 90% | 20% | 0% | 30% | 20% | 30% | 80% | 10% | Tier 2 |
Jewellery & Luxury and Real Estate both record a mean recommended tier of 2.90 — the joint highest. Hospitality follows at 2.80. Human Resources and Retail each score 2.50, with Human Resources showing 70% Antibot and 60% CAPTCHA deployment.
Adult, Construction, and Events anchor the most accessible sectors
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Consumer Electronics | 80% | 10% | 10% | 30% | 10% | 50% | 100% | 20% | Tier 2 |
| Military | 70% | 0% | 10% | 20% | 0% | 10% | 60% | 0% | Tier 2 |
| Transport & Logistics | 80% | 50% | 10% | 30% | 0% | 20% | 90% | 0% | Tier 2 |
| Mail & Delivery | 80% | 40% | 10% | 20% | 0% | 20% | 90% | 0% | Tier 2 |
| Restaurants | 90% | 50% | 30% | 70% | 40% | 10% | 50% | 20% | Tier 2 |
| Wholesalers | 80% | 20% | 40% | 60% | 30% | 30% | 70% | 10% | Tier 2 |
| Wine & Spirits | 100% | 30% | 10% | 50% | 20% | 0% | 70% | 30% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Manufacturing | 80% | 10% | 20% | 40% | 0% | 40% | 80% | 0% | Tier 2 |
| Maritime | 90% | 20% | 0% | 30% | 10% | 30% | 80% | 10% | Tier 2 |
| Reference | 80% | 50% | 20% | 30% | 10% | 10% | 60% | 40% | Tier 2 |
| Textiles & Nonwovens | 100% | 0% | 10% | 30% | 10% | 20% | 90% | 20% | Tier 2 |
| Adult | 100% | 40% | 30% | 80% | 10% | 20% | 90% | 10% | Tier 2 |
| Construction | 70% | 10% | 30% | 30% | 20% | 10% | 50% | 10% | Tier 2 |
| Events | 100% | 10% | 10% | 100% | 30% | 0% | 70% | 0% | Tier 2 |
Adult, Construction, and Events each record the lowest mean tier at 1.90. Events is notable for 100% JavaScript deployment at that tier. Twelve sectors above share mean tiers of 2.00 to 2.20, with five sectors clustering at the 2.00 floor.