Overview
JavaScript is Sri Lanka's most common secondary barrier
| Value | |
|---|---|
| WAF | 92.3% |
| Antibot | 20.9% |
| CAPTCHA | 22.5% |
| JavaScript | 42.2% |
| Rate Limiting | 12.6% |
| TLS | 19.1% |
WAF reaches 92.3% of Sri Lanka sites. JavaScript leads secondary barriers at 42.2%. CAPTCHA follows at 22.5%, ahead of Antibot at 20.9%, TLS at 19.1%, and Rate Limiting at 12.6%.
Barrier Landscape
JavaScript is the leading secondary barrier
| Value | |
|---|---|
| WAF | 92.3% |
| Antibot | 20.9% |
| CAPTCHA | 22.5% |
| JavaScript | 42.2% |
| Rate Limiting | 12.6% |
| TLS | 19.1% |
| Has robots.txt | 79.6% |
| Blocks AI crawlers | 13.8% |
WAF is present on 92.3% of Sri Lanka domains. JavaScript rendering leads secondary barriers at 42.2%. CAPTCHA follows at 22.5%, Antibot at 20.9%, TLS at 19.1%, and Rate Limiting at 12.6%.
Two-barrier sites are Sri Lanka's most common configuration
| % of sites | |
|---|---|
| 0 barriers | 5.1% |
| 1 barrier | 29.3% |
| 2 barriers | 34.8% |
| 3 barriers | 16.9% |
| 4 barriers | 9.8% |
| 5 barriers | 3.7% |
| 6 barriers | 0.4% |
Exactly two barriers is the most common configuration at 34.8%, followed by one-barrier sites at 29.3%. 5.1% of Sri Lanka domains have no barriers — the highest zero-barrier share in this group. 30.8% deploy three or more.
Most Sri Lanka sites are Easy to access
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
Across 1,102 Sri Lanka domains, the recommended Zyte browser tier has a median of 2 and a mean of 1.70. Tier 1 (46.0%) and Tier 2 (41.2%) are closely matched, the tightest split in this group.
Simple and Easy sites are nearly equal in share
| % of sites | |
|---|---|
| 1 - Simple | 46% |
| 2 - Easy | 41.2% |
| 3 - Moderate | 9.9% |
| 4 - Complex | 2.1% |
| 5 - Advanced | 0.8% |
Tier 1 covers 46.0% of Sri Lanka sites; Tier 2 is close behind at 41.2%, totalling 87.2% within the two simplest tiers. Tier 3 accounts for 9.9%; Tiers 4 and 5 together reach just 2.9%.
robots.txt
AI crawlers top Sri Lanka's robots.txt bot categories
| % of sites | |
|---|---|
| AI crawlers | 15.6% |
| SEO / search indexing | 12.3% |
| SEO audit tools | 14% |
| Commercial scrapers | 7.8% |
| Social / content scrapers | 6.3% |
| Archive / research crawlers | 4.4% |
| Monitoring / uptime bots | 0.5% |
AI crawlers are named in 15.6% of all Sri Lanka sites. SEO audit tools follow at 14.0%, ahead of SEO/search indexing at 12.3%. Monitoring bots appear in just 0.5% of sites.
Uniform rules are Sri Lanka's most common robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 41.3% |
| Path-level mixed rules for same agent | 33% |
| Allow-only (no disallow rules) | 2.4% |
| Mixed AI posture (block some, allow others) | 2.5% |
| Blanket Disallow: / | 2.7% |
Uniform per-agent rules apply to 41.3% of all Sri Lanka sites. Path-level mixed rules cover 33.0%. The 20.4% of sites without a robots.txt file have no crawl instructions at all.
GPTBot faces the most Disallow rules in Sri Lanka's robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.2% | 2.5% |
| CCBot | 8.6% | 0.9% |
| ClaudeBot | 8.3% | 1.3% |
| Google-Extended | 8.3% | 1.5% |
| Bytespider | 7.5% | 0.4% |
| meta-externalagent | 3.3% | — |
| ChatGPT-User | 5.8% | 2.7% |
| PerplexityBot | 6% | 2.4% |
| Applebot-Extended | 5.1% | 0.4% |
| OAI-SearchBot | 4.4% | 2.3% |
| anthropic-ai | 5.1% | 0.7% |
GPTBot carries the highest Disallow rate at 11.2% of all Sri Lanka sites. ChatGPT-User receives the most explicit Allow directives at 2.7%, ahead of PerplexityBot at 2.4%.
Visual Art joins six other sectors at full named-bot restriction
| % of sites | |
|---|---|
| Civil Engineering | 100% |
| Consumer Electronics | 100% |
| Design | 100% |
| Entertainment | 100% |
| Retail | 100% |
| Telecom | 100% |
| Visual Art | 100% |
| Adult | 90% |
| Automotive | 90% |
| Beauty & Cosmetics | 90% |
| Comics & Animation | 90% |
| Computer & Video Games | 90% |
| General | 90% |
| Healthcare | 90% |
| Insurance | 90% |
Civil Engineering, Consumer Electronics, Design, Entertainment, Retail, Telecom, and Visual Art each show 100% of their robots.txt sites naming at least one bot to restrict. A further eight sectors sit at 90%.
Airlines and Telecom lead explicit bot Allow rules
| % of sites | |
|---|---|
| Airlines | 80% |
| Telecom | 80% |
| Architecture | 70% |
| Photography | 70% |
| Public Policy | 70% |
| Publishing | 70% |
| Retail | 70% |
| Science | 70% |
| Facilities Services | 60% |
| Market Research | 60% |
| Non-Profit | 60% |
| Software & Dev | 60% |
| Travel & Tourism | 60% |
| Veterinary | 60% |
| Aerospace & Defence | 50% |
Airlines and Telecom top the allow list with 80% of their robots.txt sites naming at least one bot in an Allow directive. Architecture, Photography, Public Policy, Publishing, Retail, and Science follow at 70%.
Industry Patterns
Jewellery & Luxury and Real Estate jointly top scraping difficulty
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Jewellery & Luxury | 100% | 20% | 0% | 40% | 50% | 60% | 70% | 10% | Tier 3 |
| Real Estate | 100% | 20% | 20% | 50% | 50% | 10% | 60% | 30% | Tier 3 |
| Furniture | 100% | 60% | 20% | 40% | 10% | 70% | 70% | 10% | Tier 3 |
| Hospitality | 90% | 30% | 0% | 60% | 20% | 40% | 60% | 10% | Tier 3 |
| Equipment & Supplies | 90% | 20% | 20% | 40% | 20% | 0% | 70% | 20% | Tier 3 |
| Medical Devices | 100% | 20% | 10% | 50% | 20% | 0% | 90% | 0% | Tier 2 |
| Apparel & Fashion | 90% | 10% | 20% | 50% | 10% | 30% | 80% | 10% | Tier 2 |
| Retail | 100% | 10% | 0% | 70% | 10% | 10% | 100% | 50% | Tier 2 |
| Human Resources | 100% | 80% | 60% | 50% | 40% | 20% | 70% | 10% | Tier 2 |
| Insurance | 90% | 10% | 20% | 30% | 0% | 30% | 90% | 0% | Tier 2 |
| Beauty & Cosmetics | 80% | 20% | 10% | 30% | 30% | 40% | 90% | 10% | Tier 2 |
| Computer Hardware | 90% | 20% | 0% | 30% | 20% | 20% | 70% | 10% | Tier 2 |
| Food & Beverages | 90% | 30% | 20% | 50% | 20% | 0% | 70% | 30% | Tier 2 |
| Restaurants | 90% | 30% | 30% | 50% | 40% | 20% | 50% | 10% | Tier 2 |
| Wine & Spirits | 100% | 40% | 10% | 50% | 20% | 10% | 80% | 30% | Tier 2 |
Jewellery & Luxury and Real Estate share the highest mean recommended tier at 2.80. Furniture, Hospitality, and Medical Devices all follow at 2.60. Furniture is notable for 70% TLS adoption and 60% antibot coverage.
Automotive and Chemicals share the most accessible floor
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Consumer Electronics | 80% | 10% | 20% | 40% | 10% | 40% | 100% | 20% | Tier 2 |
| Customer Services | 100% | 20% | 30% | 40% | 30% | 20% | 70% | 0% | Tier 2 |
| Maritime | 90% | 10% | 10% | 30% | 10% | 30% | 90% | 10% | Tier 2 |
| Banking | 100% | 30% | 10% | 30% | 10% | 20% | 70% | 0% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Mail & Delivery | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Public Safety | 100% | 20% | 30% | 60% | 20% | 40% | 90% | 10% | Tier 2 |
| Sporting Goods | 90% | 10% | 20% | 30% | 30% | 20% | 70% | 0% | Tier 2 |
| Transport & Logistics | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Manufacturing | 80% | 10% | 10% | 30% | 0% | 40% | 70% | 0% | Tier 2 |
| Recruitment & Staffing | 100% | 50% | 60% | 40% | 10% | 30% | 70% | 20% | Tier 2 |
| TV & Film | 100% | 60% | 10% | 70% | 40% | 10% | 50% | 10% | Tier 2 |
| Visual Art | 100% | 30% | 20% | 30% | 0% | 20% | 100% | 20% | Tier 2 |
| Automotive | 100% | 10% | 0% | 20% | 30% | 20% | 90% | 20% | Tier 2 |
| Chemicals | 60% | 10% | 10% | 40% | 0% | 20% | 70% | 10% | Tier 2 |
Automotive and Chemicals each record a mean recommended tier of 1.80, the lowest in the bottom-industries group. Manufacturing, Recruitment & Staffing, TV & Film, and Visual Art sit just above at 1.90.