Overview
JavaScript is Yemen's most common secondary barrier
| Value | |
|---|---|
| WAF | 92.4% |
| Antibot | 20.4% |
| CAPTCHA | 24.8% |
| JavaScript | 39.5% |
| Rate Limiting | 12.6% |
| TLS | 18.3% |
WAF covers 92.4% of Yemen sites. Among secondary barriers, JavaScript is the most widely adopted at 39.5%, ahead of CAPTCHA at 24.8%, Antibot at 20.4%, and TLS at 18.3%.
Barrier Landscape
CAPTCHA and Antibot are the strongest secondary barriers
| Value | |
|---|---|
| WAF | 92.4% |
| Antibot | 20.4% |
| CAPTCHA | 24.8% |
| JavaScript | 39.5% |
| Rate Limiting | 12.6% |
| TLS | 18.3% |
| Has robots.txt | 80.4% |
| Blocks AI crawlers | 13.3% |
After WAF at 92.4%, JavaScript leads secondary barriers at 39.5%. CAPTCHA (24.8%), Antibot (20.4%), and TLS (18.3%) follow. Rate Limiting at 12.6% is the least common.
Single-barrier sites are the most common configuration
| % of sites | |
|---|---|
| 0 barriers | 5.2% |
| 1 barrier | 30.1% |
| 2 barriers | 34.6% |
| 3 barriers | 16.6% |
| 4 barriers | 9.3% |
| 5 barriers | 3.7% |
| 6 barriers | 0.5% |
34.6% of Yemen sites deploy exactly two barriers — the modal count. Only 5.2% deploy none, while 29.6% stack three or more.
Most Yemen sites are Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.6 |
The median recommended tier is 1, meaning more than half of Yemen sites are accessible with basic HTTP retrieval. The mean of 1.64 reflects a tail of harder sites pushing the average up.
Most Yemen sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 52.6% |
| 2 - Easy | 35.1% |
| 3 - Moderate | 9.4% |
| 4 - Complex | 2% |
| 5 - Advanced | 0.9% |
52.6% of Yemen sites require only Tier 1 retrieval. A further 35.1% need browser rendering at Tier 2. The remaining 12.3% split across Tiers 3, 4, and 5, with fewer than 1% at the hardest level.
robots.txt
AI crawlers are the most-cited bot category in robots.txt
| % of sites | |
|---|---|
| AI crawlers | 15% |
| SEO / search indexing | 12.4% |
| SEO audit tools | 12% |
| Commercial scrapers | 7.4% |
| Social / content scrapers | 6% |
| Archive / research crawlers | 4.8% |
| Monitoring / uptime bots | 0.4% |
15.0% of all Yemen sites name at least one AI crawler in their robots.txt — the largest single category. SEO/search indexing (12.4%) and SEO audit tools (12.0%) are close behind. Monitoring bots are named by just 0.4%.
Uniform per-agent rules are the dominant robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 41% |
| Path-level mixed rules for same agent | 33.9% |
| Allow-only (no disallow rules) | 2.4% |
| Mixed AI posture (block some, allow others) | 2% |
| Blanket Disallow: / | 3% |
41.0% of Yemen sites use uniform per-agent rules with no path mixing. Path-level mixed rules account for 33.9%. Sites with no robots.txt at all make up 19.6% of the sample.
GPTBot faces the most Disallow rules in Yemen's robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.4% | 1.8% |
| CCBot | 8.5% | 1.2% |
| ClaudeBot | 8.2% | 1.2% |
| Google-Extended | 8.2% | 1.4% |
| Bytespider | 7.4% | 0.7% |
| meta-externalagent | 3% | 0.2% |
| ChatGPT-User | 5.6% | 2.3% |
| PerplexityBot | 5.4% | 1.9% |
| Applebot-Extended | 5.1% | 0.7% |
| OAI-SearchBot | 3.2% | 1.8% |
| anthropic-ai | 5% | 0.8% |
GPTBot is named in a Disallow rule on 11.4% of all Yemen sites — the most blocked AI agent. ChatGPT-User leads explicit Allow rules at 2.3%, the highest allow rate of any AI agent.
Four sectors have every site naming at least one bot to restrict
| % of sites | |
|---|---|
| Design | 100% |
| Entertainment | 100% |
| Foreign Language | 100% |
| Telecom | 100% |
| Adult | 90% |
| Apparel & Fashion | 90% |
| Automotive | 90% |
| Biotech | 90% |
| Consulting | 90% |
| Consumer Electronics | 90% |
| Crafts | 90% |
| General | 90% |
| Healthcare | 90% |
| Manufacturing | 90% |
| Mass Media | 90% |
Design, Entertainment, Foreign Language, and Telecom each show a 100% named-bot Disallow rate — every sampled site in these sectors names at least one bot to restrict. Nine further industries reach 90%.
Eight sectors share the top explicit bot Allow rate
| % of sites | |
|---|---|
| Architecture | 70% |
| Photography | 70% |
| Publishing | 70% |
| Retail | 70% |
| Science | 70% |
| Telecom | 70% |
| Travel & Tourism | 70% |
| Wholesalers | 70% |
| Facilities Services | 66.7% |
| Airlines | 60% |
| Education | 60% |
| Equipment & Supplies | 60% |
| Foreign Language | 60% |
| IT Services | 60% |
| Manufacturing | 60% |
Architecture, Photography, Publishing, Retail, Science, Telecom, Travel & Tourism, and Wholesalers all show 70% of their sites naming at least one bot in an explicit Allow rule — the highest rate recorded across Yemen industries.
Industry Patterns
Computer Hardware is the hardest sector to access programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Computer Hardware | 90% | 10% | 0% | 30% | 10% | 30% | 100% | 10% | Tier 3 |
| Beauty & Cosmetics | 100% | 20% | 20% | 70% | 20% | 10% | 80% | 20% | Tier 3 |
| Hospitality | 100% | 20% | 20% | 60% | 10% | 40% | 70% | 0% | Tier 3 |
| Public Policy | 100% | 33.3% | 33.3% | 16.7% | 16.7% | 0% | 83.3% | 0% | Tier 2 |
| Retail | 100% | 10% | 0% | 70% | 20% | 10% | 90% | 40% | Tier 2 |
| Automotive | 100% | 10% | 30% | 50% | 10% | 20% | 100% | 20% | Tier 2 |
| Airlines | 90% | 10% | 10% | 40% | 30% | 10% | 90% | 20% | Tier 2 |
| Consumer Electronics | 70% | 30% | 30% | 40% | 10% | 60% | 100% | 10% | Tier 2 |
| Human Resources | 100% | 60% | 60% | 40% | 60% | 30% | 70% | 10% | Tier 2 |
| Real Estate | 90% | 50% | 30% | 60% | 40% | 20% | 60% | 0% | Tier 2 |
| Apparel & Fashion | 100% | 20% | 30% | 70% | 20% | 30% | 90% | 10% | Tier 2 |
| Wholesalers | 80% | 20% | 30% | 50% | 30% | 40% | 80% | 10% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Maritime | 80% | 30% | 0% | 40% | 10% | 30% | 70% | 10% | Tier 2 |
| Reference | 80% | 40% | 30% | 30% | 10% | 10% | 70% | 40% | Tier 2 |
Computer Hardware records the highest mean recommended tier at 2.50. Public Policy and Retail share the same mean. All top-15 industries cluster between tier means of 2.00 and 2.50, a tight difficulty band.
Museums, Libraries, and Mail & Delivery are the most accessible sectors
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Travel & Tourism | 100% | 30% | 10% | 70% | 30% | 10% | 80% | 30% | Tier 2 |
| Pharmaceuticals | 90% | 20% | 10% | 30% | 10% | 10% | 70% | 20% | Tier 2 |
| Adult | 100% | 40% | 30% | 80% | 10% | 20% | 90% | 10% | Tier 2 |
| Education | 100% | 20% | 10% | 30% | 10% | 10% | 80% | 10% | Tier 2 |
| Finance | 80% | 10% | 0% | 10% | 20% | 20% | 60% | 20% | Tier 2 |
| Food & Beverages | 100% | 40% | 10% | 80% | 20% | 10% | 70% | 20% | Tier 2 |
| Jewellery & Luxury | 100% | 40% | 20% | 50% | 50% | 30% | 70% | 10% | Tier 2 |
| Manufacturing | 90% | 30% | 20% | 30% | 0% | 50% | 90% | 10% | Tier 2 |
| Transport & Logistics | 80% | 30% | 20% | 10% | 10% | 40% | 70% | 0% | Tier 2 |
| Plastics & Polymers | 83.3% | 0% | 0% | 16.7% | 0% | 16.7% | 50% | 0% | Tier 2 |
| Advertising & Mktg | 100% | 30% | 60% | 40% | 0% | 30% | 90% | 10% | Tier 2 |
| Healthcare | 100% | 10% | 10% | 30% | 0% | 10% | 100% | 40% | Tier 2 |
| Libraries | 100% | 10% | 20% | 50% | 0% | 0% | 80% | 0% | Tier 2 |
| Mail & Delivery | 80% | 40% | 30% | 20% | 10% | 20% | 80% | 0% | Tier 2 |
| Museums | 100% | 30% | 20% | 40% | 10% | 10% | 80% | 10% | Tier 2 |
Five sectors share the lowest mean tier of 1.80 among the bottom 15 industries. Seven further sectors cluster at 1.90, indicating broadly accessible scraping conditions across this group.