Overview
Newspapers joins Israel's full named-bot Disallow group
| Value | |
|---|---|
| WAF | 93.3% |
| Antibot | 20.9% |
| CAPTCHA | 21.6% |
| JavaScript | 42.7% |
| Rate Limiting | 13.3% |
| TLS | 18.7% |
WAF reaches 93.3% and JavaScript 42.7%. CAPTCHA at 21.6% is the lowest secondary barrier.
Barrier Landscape
JavaScript leads secondary barriers at well over four in ten sites
| Value | |
|---|---|
| WAF | 93.3% |
| Antibot | 20.9% |
| CAPTCHA | 21.6% |
| JavaScript | 42.7% |
| Rate Limiting | 13.3% |
| TLS | 18.7% |
| Has robots.txt | 80.2% |
| Blocks AI crawlers | 14.3% |
WAF dominates at 93.3%. JavaScript is the top secondary barrier at 42.7%. CAPTCHA follows at 21.6% and Antibot at 20.9%. TLS sits at 18.7% and Rate Limiting at 13.3%.
Two-barrier sites are Israel's most common configuration
| % of sites | |
|---|---|
| 0 barriers | 4.4% |
| 1 barrier | 28.3% |
| 2 barriers | 36.2% |
| 3 barriers | 18.9% |
| 4 barriers | 8.2% |
| 5 barriers | 3.6% |
| 6 barriers | 0.4% |
Two-barrier sites are the most common at 36.2% — the strongest modal concentration in this profile. Three or more barriers appear on 31.1% of sites. Barrier-free sites are 4.4%, the lowest zero-barrier rate in this group.
Most Israel sites are Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
A mean recommended browser tier of 1.69 and median of 2 indicate that Tier 1 accounts for a large share but fewer than half of sites. Tier 2 is the dominant recommendation for the remainder.
Nearly nine in ten sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 47.5% |
| 2 - Easy | 39.3% |
| 3 - Moderate | 10.3% |
| 4 - Complex | 2.1% |
| 5 - Advanced | 0.8% |
Tier 1 applies to 47.5% of sites and Tier 2 to 39.3%, together reaching 86.8%. Tier 3 covers a further 10.3%. Tiers 4 and 5 account for just 2.9% combined.
robots.txt
AI crawlers top Israel's robots.txt bot categories
| % of sites | |
|---|---|
| AI crawlers | 15.8% |
| SEO / search indexing | 12.6% |
| SEO audit tools | 13.4% |
| Commercial scrapers | 8.5% |
| Social / content scrapers | 6% |
| Archive / research crawlers | 4.2% |
| Monitoring / uptime bots | 0.5% |
AI crawlers appear in robots.txt on 15.8% of all 1,109 Israel sites. SEO audit tools follow at 13.4%, with SEO/search indexing bots at 12.6%. Commercial scrapers are named by 8.5% of sites.
Uniform rules are Israel's most common robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 42.2% |
| Path-level mixed rules for same agent | 32.8% |
| Allow-only (no disallow rules) | 2.4% |
| Mixed AI posture (block some, allow others) | 2.4% |
| Blanket Disallow: / | 2.6% |
Uniform per-agent rules are the dominant posture at 42.2% of all sites. Path-level mixed rules apply to 32.8%. The 19.8% of sites without any robots.txt exceed every other named posture type in volume.
GPTBot carries the heaviest block load among AI agents
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.8% | 2.3% |
| CCBot | 8.7% | 1.1% |
| ClaudeBot | 8.5% | 1.3% |
| Google-Extended | 8.3% | 1.4% |
| Bytespider | 6.9% | 0.5% |
| meta-externalagent | 2.9% | 0.2% |
| ChatGPT-User | 5.9% | 2.3% |
| PerplexityBot | 5.5% | 1.9% |
| Applebot-Extended | 4.9% | 0.5% |
| OAI-SearchBot | 4% | 1.8% |
| anthropic-ai | 5.1% | 0.7% |
11.8% of all Israel sites carry a Disallow for GPTBot, well ahead of CCBot at 8.7% and ClaudeBot at 8.5%. ChatGPT-User and GPTBot each receive the most explicit Allow rules at 2.3%.
Newspapers joins Israel's full named-bot Disallow group
| % of sites | |
|---|---|
| Computer & Video Games | 100% |
| Design | 100% |
| Newspapers | 100% |
| Visual Art | 100% |
| Adult | 90% |
| Automotive | 90% |
| Beauty & Cosmetics | 90% |
| Entertainment | 90% |
| General | 90% |
| Mass Media | 90% |
| Medical Devices | 90% |
| Online Services | 90% |
| Plastics & Polymers | 90% |
| Public Relations | 90% |
| Public Safety | 90% |
Computer & Video Games, Design, Newspapers, and Visual Art each have every measured site naming at least one bot to restrict. Newspapers at 100% is notable. Eleven more sectors — including Automotive and Mass Media — sit at 90%.
Automotive leads explicit bot Allow rules
| % of sites | |
|---|---|
| Automotive | 90% |
| Architecture | 70% |
| Photography | 70% |
| Publishing | 70% |
| Telecom | 70% |
| Farms & Ranches | 62.5% |
| Airlines | 60% |
| Civil Engineering | 60% |
| Construction | 60% |
| Graphic Design | 60% |
| Import & Export | 60% |
| Investment | 60% |
| Mental Health | 60% |
| Pharmaceuticals | 60% |
| Plastics & Polymers | 60% |
Automotive is the clear standout: 90% of its sites name at least one bot in an explicit Allow rule. Architecture, Photography, Publishing, and Telecom follow at 70%. No other sector exceeds 70% in this market.
Industry Patterns
Real Estate is the hardest sector at near-Moderate mean difficulty
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Real Estate | 100% | 10% | 10% | 60% | 40% | 20% | 40% | 20% | Tier 3 |
| Jewellery & Luxury | 100% | 10% | 0% | 30% | 30% | 50% | 60% | 10% | Tier 3 |
| Hospitality | 90% | 20% | 0% | 60% | 30% | 40% | 60% | 20% | Tier 3 |
| Furniture | 100% | 50% | 40% | 40% | 20% | 50% | 80% | 20% | Tier 3 |
| Equipment & Supplies | 90% | 40% | 20% | 50% | 20% | 10% | 70% | 10% | Tier 3 |
| Computer Hardware | 90% | 10% | 0% | 20% | 20% | 20% | 80% | 10% | Tier 2 |
| Apparel & Fashion | 100% | 10% | 20% | 70% | 20% | 20% | 80% | 10% | Tier 2 |
| Retail | 100% | 10% | 0% | 70% | 20% | 10% | 90% | 40% | Tier 2 |
| Customer Services | 100% | 20% | 30% | 70% | 40% | 30% | 60% | 0% | Tier 2 |
| Medical Devices | 100% | 0% | 20% | 50% | 20% | 0% | 100% | 10% | Tier 2 |
| Human Resources | 100% | 60% | 50% | 50% | 40% | 30% | 80% | 10% | Tier 2 |
| Restaurants | 90% | 20% | 20% | 50% | 30% | 30% | 70% | 10% | Tier 2 |
| Beauty & Cosmetics | 80% | 40% | 30% | 30% | 30% | 40% | 90% | 20% | Tier 2 |
| Consumer Electronics | 80% | 10% | 0% | 50% | 10% | 30% | 80% | 10% | Tier 2 |
| Crafts | 100% | 30% | 30% | 40% | 20% | 10% | 80% | 10% | Tier 2 |
Real Estate leads at a mean recommended tier of 2.90. Jewellery & Luxury follows at 2.80, with Hospitality at 2.70. The top five sectors all carry tier medians of 3, indicating Moderate complexity is the norm at the top.
Six sectors share the most accessible floor
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Mail & Delivery | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Maritime | 90% | 10% | 0% | 40% | 0% | 30% | 90% | 10% | Tier 2 |
| Public Safety | 100% | 40% | 50% | 60% | 20% | 30% | 90% | 20% | Tier 2 |
| Sporting Goods | 80% | 10% | 20% | 40% | 20% | 30% | 60% | 0% | Tier 2 |
| Textiles & Nonwovens | 100% | 10% | 50% | 30% | 10% | 10% | 90% | 10% | Tier 2 |
| Transport & Logistics | 70% | 50% | 10% | 20% | 0% | 20% | 80% | 0% | Tier 2 |
| Veterinary | 100% | 20% | 20% | 40% | 20% | 10% | 90% | 10% | Tier 2 |
| Food & Beverages | 100% | 40% | 20% | 40% | 20% | 0% | 80% | 30% | Tier 2 |
| Adult | 100% | 40% | 30% | 80% | 10% | 20% | 90% | 10% | Tier 2 |
| Import & Export | 100% | 30% | 40% | 40% | 0% | 40% | 80% | 10% | Tier 2 |
| Manufacturing | 80% | 10% | 10% | 30% | 0% | 40% | 70% | 0% | Tier 2 |
| Recruitment & Staffing | 90% | 40% | 60% | 20% | 0% | 20% | 80% | 30% | Tier 2 |
| Reference | 90% | 60% | 30% | 50% | 30% | 10% | 60% | 30% | Tier 2 |
| Visual Art | 100% | 30% | 20% | 30% | 0% | 20% | 100% | 20% | Tier 2 |
Visual Art, Adult, Import & Export, Manufacturing, Recruitment & Staffing, and Reference all post a mean tier of 1.90 — the floor for this sample. All carry a tier median of 2.