Overview
JavaScript is the clear leading secondary barrier
| Value | |
|---|---|
| WAF | 92.7% |
| Antibot | 20.4% |
| CAPTCHA | 22.8% |
| JavaScript | 42.3% |
| Rate Limiting | 14.4% |
| TLS | 16.3% |
WAF covers 92.7% of Mongolian sites. Among the five secondary barriers, JavaScript ranks first at 42.3%. CAPTCHA at 22.8% and Antibot at 20.4% follow as the next strongest.
Barrier Landscape
JavaScript is the clear leading secondary barrier
| Value | |
|---|---|
| WAF | 92.7% |
| Antibot | 20.4% |
| CAPTCHA | 22.8% |
| JavaScript | 42.3% |
| Rate Limiting | 14.4% |
| TLS | 16.3% |
| Has robots.txt | 79.2% |
| Blocks AI crawlers | 14.1% |
WAF leads at 92.7%. JavaScript at 42.3% is the strongest secondary barrier. CAPTCHA follows at 22.8%, with Antibot, TLS, and Rate Limiting completing the profile.
Two-barrier sites are Mongolia's most common configuration
| % of sites | |
|---|---|
| 0 barriers | 4.8% |
| 1 barrier | 30.4% |
| 2 barriers | 34.8% |
| 3 barriers | 16.5% |
| 4 barriers | 8.8% |
| 5 barriers | 4.4% |
| 6 barriers | 0.4% |
Two-barrier sites account for 34.8% of Mongolian domains — the single largest group. Sites with three or more barriers total 30.1%. Barrier-free sites are rare at 4.8%.
Most Mongolia sites are Easy to access
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
The browser-based recommended tier averages 1.71 across 1,084 Mongolian sites. The median of 2 means fewer than half require only Tier 1 retrieval.
Nearly nine in ten sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 46.1% |
| 2 - Easy | 41.1% |
| 3 - Moderate | 9.6% |
| 4 - Complex | 2.5% |
| 5 - Advanced | 0.7% |
Tier 1 (Simple) applies to 46.1% of sites; Tier 2 (Easy) to a further 41.1%. Tier 3 accounts for 9.6%, while Tier 4 and 5 combined reach just 3.2%.
robots.txt
AI crawlers are the most-cited bot category in robots.txt
| % of sites | |
|---|---|
| AI crawlers | 15.7% |
| SEO / search indexing | 12.5% |
| SEO audit tools | 12.5% |
| Commercial scrapers | 8.4% |
| Social / content scrapers | 6.5% |
| Archive / research crawlers | 4.1% |
| Monitoring / uptime bots | 0.5% |
AI crawlers are named in 15.7% of all Mongolian sites. SEO / search indexing and SEO audit tools each appear at 12.5%, sharing second place across 1,084 sites.
Uniform per-agent rules are the dominant robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 40.8% |
| Path-level mixed rules for same agent | 32.7% |
| Allow-only (no disallow rules) | 2.6% |
| Mixed AI posture (block some, allow others) | 2.6% |
| Blanket Disallow: / | 2.9% |
Uniform per-agent rules appear on 40.8% of all Mongolian sites. Path-level mixed rules account for 32.7%. Around 20.8% of sites publish no robots.txt at all. Blanket Disallow covers 2.9%.
GPTBot faces the most Disallow rules in Mongolia's robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.5% | 1.9% |
| CCBot | 9% | 0.9% |
| ClaudeBot | 8.7% | 0.8% |
| Google-Extended | 8.6% | 1.5% |
| Bytespider | 7.3% | 0.5% |
| meta-externalagent | 3.1% | 0.2% |
| ChatGPT-User | 6% | 2.9% |
| PerplexityBot | 6% | 2.1% |
| Applebot-Extended | 5.1% | 0.4% |
| OAI-SearchBot | 4.3% | 2.2% |
| anthropic-ai | 5.6% | 0.8% |
GPTBot carries the highest Disallow rate at 11.5% of all Mongolian sites. ChatGPT-User leads on explicit Allow rules at 2.9%, ahead of OAI-SearchBot at 2.2%.
Six sectors reach universal named-bot restriction
| % of sites | |
|---|---|
| Design | 100% |
| Entertainment | 100% |
| Medical Devices | 100% |
| Newspapers | 100% |
| Printing | 100% |
| Veterinary | 100% |
| Accounting & Auditing | 90% |
| Adult | 90% |
| Ceramics & Pottery | 90% |
| Comics & Animation | 90% |
| Computer & Video Games | 90% |
| Consulting | 90% |
| Consumer Electronics | 90% |
| General | 90% |
| Healthcare | 90% |
Design, Entertainment, Medical Devices, Newspapers, Printing, and Veterinary all show 100% of their sites naming at least one bot to restrict. Accounting & Auditing and several others follow at 90%.
Market Research leads explicit bot Allow rules
| % of sites | |
|---|---|
| Market Research | 80% |
| Airlines | 70% |
| Automotive | 70% |
| Photography | 70% |
| Publishing | 70% |
| Retail | 70% |
| Science | 70% |
| Architecture | 60% |
| Graphic Design | 60% |
| Non-Profit | 60% |
| Outsourcing | 60% |
| Public Policy | 60% |
| Real Estate | 60% |
| Software & Dev | 60% |
| Telecom | 60% |
Market Research sites name at least one bot in an Allow rule on 80% of domains. Airlines, Automotive, Photography, Publishing, Retail, and Science follow at 70%.
Industry Patterns
Jewellery & Luxury is the hardest sector to access programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Jewellery & Luxury | 100% | 20% | 20% | 30% | 60% | 50% | 50% | 0% | Tier 3 |
| Real Estate | 100% | 20% | 20% | 50% | 50% | 20% | 60% | 20% | Tier 3 |
| Retail | 100% | 10% | 0% | 70% | 20% | 10% | 90% | 50% | Tier 3 |
| Human Resources | 100% | 60% | 60% | 30% | 50% | 30% | 70% | 10% | Tier 3 |
| Hospitality | 80% | 30% | 0% | 50% | 20% | 40% | 60% | 10% | Tier 2 |
| Automotive | 100% | 30% | 0% | 60% | 50% | 40% | 70% | 20% | Tier 2 |
| Equipment & Supplies | 90% | 20% | 20% | 60% | 20% | 0% | 70% | 20% | Tier 2 |
| Furniture | 100% | 50% | 30% | 40% | 10% | 50% | 70% | 0% | Tier 2 |
| Medical Devices | 90% | 10% | 10% | 40% | 20% | 20% | 100% | 10% | Tier 2 |
| Sporting Goods | 80% | 20% | 20% | 20% | 40% | 30% | 60% | 0% | Tier 2 |
| Computer Hardware | 100% | 20% | 0% | 20% | 20% | 20% | 90% | 10% | Tier 2 |
| Beauty & Cosmetics | 90% | 40% | 30% | 50% | 50% | 20% | 80% | 10% | Tier 2 |
| Recruitment & Staffing | 90% | 40% | 60% | 0% | 10% | 40% | 80% | 30% | Tier 2 |
| Restaurants | 80% | 30% | 10% | 60% | 50% | 20% | 70% | 10% | Tier 2 |
| Wholesalers | 80% | 10% | 40% | 60% | 30% | 30% | 80% | 10% | Tier 2 |
Jewellery & Luxury records the highest mean recommended tier at 3.00, with a median of 3 (Moderate). Real Estate at 2.70 and Retail at 2.60 follow as the next most demanding sectors.
Plastics & Polymers, Manufacturing, and Adult are the most accessible sectors
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Apparel & Fashion | 100% | 20% | 30% | 70% | 30% | 30% | 80% | 10% | Tier 2 |
| Consumer Electronics | 80% | 10% | 20% | 50% | 10% | 40% | 90% | 20% | Tier 2 |
| Mail & Delivery | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Public Safety | 100% | 20% | 30% | 60% | 20% | 30% | 90% | 10% | Tier 2 |
| Transport & Logistics | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Wine & Spirits | 100% | 40% | 20% | 70% | 40% | 0% | 80% | 50% | Tier 2 |
| Construction | 80% | 10% | 20% | 50% | 30% | 0% | 40% | 0% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Insurance | 80% | 10% | 10% | 20% | 0% | 20% | 90% | 0% | Tier 2 |
| Reference | 90% | 60% | 30% | 40% | 20% | 10% | 60% | 30% | Tier 2 |
| Veterinary | 100% | 20% | 30% | 40% | 10% | 0% | 100% | 10% | Tier 2 |
| Food & Beverages | 100% | 40% | 20% | 70% | 40% | 0% | 80% | 40% | Tier 2 |
| Adult | 100% | 30% | 30% | 80% | 10% | 10% | 90% | 10% | Tier 2 |
| Manufacturing | 80% | 10% | 20% | 50% | 0% | 40% | 80% | 0% | Tier 2 |
| Plastics & Polymers | 90% | 0% | 30% | 40% | 0% | 20% | 80% | 0% | Tier 2 |
The lowest mean recommended tier — 1.90 — is shared by Plastics & Polymers, Manufacturing, and Adult. Food & Beverages at 1.95 sits just above the floor, with most sectors clustered around 2.00.