Overview
JavaScript leads secondary barriers, with CAPTCHA and Antibot following
| Value | |
|---|---|
| WAF | 92.6% |
| Antibot | 20.8% |
| CAPTCHA | 22% |
| JavaScript | 42.7% |
| Rate Limiting | 14.4% |
| TLS | 17.5% |
WAF covers 92.6% of Malaysian sites. JavaScript at 42.7% leads secondary barriers. CAPTCHA at 22.0% ranks second, with Antibot at 20.8% close behind.
Barrier Landscape
JavaScript is the strongest secondary barrier
| Value | |
|---|---|
| WAF | 92.6% |
| Antibot | 20.8% |
| CAPTCHA | 22% |
| JavaScript | 42.7% |
| Rate Limiting | 14.4% |
| TLS | 17.5% |
| Has robots.txt | 78.2% |
| Blocks AI crawlers | 13.4% |
WAF leads at 92.6%. JavaScript at 42.7% is the top secondary barrier. SEO audit tools rank second by bot-category but CAPTCHA at 22.0% follows among access barriers, with Rate Limiting at 14.4%.
Nearly one in three sites stack three or more barriers
| % of sites | |
|---|---|
| 0 barriers | 5.2% |
| 1 barrier | 30% |
| 2 barriers | 33% |
| 3 barriers | 18.3% |
| 4 barriers | 8.7% |
| 5 barriers | 4.4% |
| 6 barriers | 0.4% |
Two-barrier sites are modal at 33.0%. Sites with three or more barriers total 31.8% — one of the higher stacking rates in this group. Barrier-free sites stand at 5.2%.
Most Malaysia sites are Simple to access
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
The browser-based recommended tier averages 1.70 across 1,106 Malaysian sites. A median of 2 means fewer than half require only Tier 1 retrieval; the majority land at Tier 2 (Easy).
Nearly nine in ten sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 46.8% |
| 2 - Easy | 40.3% |
| 3 - Moderate | 9.9% |
| 4 - Complex | 2.1% |
| 5 - Advanced | 0.8% |
Tier 1 (Simple) applies to 46.8% of Malaysian sites; Tier 2 (Easy) to 40.3%. Tier 3 reaches 9.9% — slightly above the regional norm — while Tier 4 and 5 together account for 2.9%.
robots.txt
AI crawlers top Malaysia's robots.txt bot categories
| % of sites | |
|---|---|
| AI crawlers | 15.2% |
| SEO / search indexing | 11.8% |
| SEO audit tools | 12.9% |
| Commercial scrapers | 8.3% |
| Social / content scrapers | 5.9% |
| Archive / research crawlers | 4.5% |
| Monitoring / uptime bots | 0.3% |
AI crawlers are named on 15.2% of all Malaysian sites. SEO audit tools appear on 12.9% — ahead of SEO / search indexing at 11.8%. Commercial scrapers follow at 8.3% across 1,106 surveyed sites.
Uniform per-agent rules dominate the robots.txt posture landscape
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 41.5% |
| Path-level mixed rules for same agent | 31.3% |
| Allow-only (no disallow rules) | 2.7% |
| Mixed AI posture (block some, allow others) | 2.2% |
| Blanket Disallow: / | 2.6% |
Uniform per-agent rules appear on 41.5% of all Malaysian sites. Path-level mixed rules follow at 31.3%. Approximately 21.8% of sites publish no robots.txt. Blanket Disallow covers 2.6%.
GPTBot is the most blocked AI agent
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.2% | 2.4% |
| CCBot | 8.7% | 1% |
| ClaudeBot | 8.2% | 1.2% |
| Google-Extended | 8% | 1.4% |
| Bytespider | 7% | 0.4% |
| meta-externalagent | 3% | 0.1% |
| ChatGPT-User | 6.1% | 2.6% |
| PerplexityBot | 5.9% | 1.9% |
| Applebot-Extended | 5% | 0.3% |
| OAI-SearchBot | 4.4% | 1.7% |
| anthropic-ai | 5.6% | 0.9% |
GPTBot appears in a Disallow rule on 11.2% of all Malaysian sites — the highest of any AI agent. ChatGPT-User leads Allow rules at 2.6%, followed by PerplexityBot at 1.9%.
Multiple Malaysia sectors all name at least one bot to restrict
| % of sites | |
|---|---|
| Computer & Video Games | 100% |
| Design | 100% |
| Entertainment | 100% |
| Retail | 100% |
| Adult | 90% |
| Agriculture | 90% |
| Beauty & Cosmetics | 90% |
| Civil Engineering | 90% |
| Comics & Animation | 90% |
| Consumer Electronics | 90% |
| Fishery | 90% |
| Food & Beverages | 90% |
| General | 90% |
| Mass Media | 90% |
| Newspapers | 90% |
Four sectors show 100% of sites naming at least one bot to restrict: Computer & Video Games, Design, Entertainment, and Retail. Adult, Agriculture, Beauty & Cosmetics, and eight further sectors follow at 90%.
Malaysia's Airlines sites lead in explicitly allowing crawlers
| % of sites | |
|---|---|
| Airlines | 80% |
| Architecture | 70% |
| Photography | 70% |
| Publishing | 70% |
| Real Estate | 70% |
| Retail | 70% |
| Travel & Tourism | 70% |
| Construction | 60% |
| Graphic Design | 60% |
| Investment | 60% |
| Mental Health | 60% |
| Oil & Gas | 60% |
| Renewable Energy | 60% |
| Science | 60% |
| Software & Dev | 60% |
Airlines sites name at least one bot in an Allow rule on 80% of domains. Architecture, Photography, Publishing, Real Estate, Retail, and Travel & Tourism all follow at 70%.
Industry Patterns
Jewellery & Luxury is the hardest sector to access programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Jewellery & Luxury | 100% | 20% | 20% | 40% | 60% | 50% | 50% | 0% | Tier 3 |
| Hospitality | 100% | 30% | 0% | 70% | 30% | 30% | 60% | 0% | Tier 3 |
| Equipment & Supplies | 90% | 40% | 10% | 50% | 40% | 0% | 60% | 20% | Tier 3 |
| Retail | 100% | 10% | 0% | 70% | 10% | 10% | 100% | 50% | Tier 2 |
| Human Resources | 100% | 80% | 60% | 50% | 40% | 20% | 70% | 10% | Tier 2 |
| Medical Devices | 100% | 10% | 10% | 40% | 20% | 0% | 90% | 0% | Tier 2 |
| Real Estate | 90% | 10% | 10% | 50% | 30% | 10% | 70% | 30% | Tier 2 |
| Beauty & Cosmetics | 90% | 30% | 30% | 30% | 20% | 20% | 90% | 10% | Tier 2 |
| Computer Hardware | 90% | 20% | 0% | 30% | 20% | 20% | 70% | 10% | Tier 2 |
| Customer Services | 100% | 20% | 30% | 50% | 30% | 30% | 60% | 0% | Tier 2 |
| Oil & Gas | 80% | 0% | 10% | 50% | 0% | 10% | 90% | 10% | Tier 2 |
| Apparel & Fashion | 100% | 20% | 30% | 70% | 20% | 20% | 80% | 10% | Tier 2 |
| Automotive | 90% | 20% | 0% | 30% | 50% | 20% | 60% | 20% | Tier 2 |
| Public Safety | 100% | 30% | 40% | 60% | 20% | 40% | 90% | 10% | Tier 2 |
| Recruitment & Staffing | 90% | 50% | 60% | 20% | 10% | 40% | 80% | 20% | Tier 2 |
Jewellery & Luxury records the highest mean recommended tier at 2.80 with a median of 3 (Moderate). Hospitality follows at 2.70, also with a median of 3. Equipment & Supplies rounds out the top three at 2.60.
Five sectors share the most accessible floor
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Wholesalers | 80% | 10% | 30% | 50% | 30% | 30% | 70% | 10% | Tier 2 |
| Food & Beverages | 100% | 60% | 40% | 70% | 40% | 0% | 90% | 50% | Tier 2 |
| Consumer Electronics | 80% | 10% | 0% | 40% | 10% | 40% | 90% | 10% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Mail & Delivery | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Restaurants | 90% | 40% | 30% | 60% | 30% | 10% | 80% | 10% | Tier 2 |
| Textiles & Nonwovens | 100% | 0% | 30% | 30% | 0% | 20% | 100% | 10% | Tier 2 |
| Banking | 90% | 20% | 10% | 40% | 20% | 10% | 50% | 0% | Tier 2 |
| Crafts | 100% | 30% | 20% | 60% | 20% | 10% | 70% | 20% | Tier 2 |
| Furniture | 100% | 40% | 30% | 40% | 20% | 20% | 70% | 10% | Tier 2 |
| Manufacturing | 80% | 10% | 10% | 30% | 0% | 40% | 70% | 0% | Tier 2 |
| Maritime | 90% | 10% | 0% | 40% | 10% | 30% | 90% | 10% | Tier 2 |
| Reference | 90% | 60% | 30% | 50% | 30% | 10% | 60% | 30% | Tier 2 |
| Renewable Energy | 90% | 0% | 10% | 50% | 10% | 20% | 90% | 0% | Tier 2 |
| Sporting Goods | 70% | 10% | 20% | 30% | 30% | 40% | 60% | 0% | Tier 2 |
Manufacturing, Maritime, Reference, Renewable Energy, and Sporting Goods each record a mean recommended tier of 1.90 — the lowest in Malaysia. Banking and Crafts follow at 1.90 alongside them, with Furniture also at 1.90.