Overview
JavaScript and TLS lead the secondary barrier mix
| Value | |
|---|---|
| WAF | 93.1% |
| Antibot | 21.6% |
| CAPTCHA | 21.5% |
| JavaScript | 43.5% |
| Rate Limiting | 13.8% |
| TLS | 19.9% |
WAF is deployed by 93.1% of sites. JavaScript leads secondary barriers at 43.5%. TLS is notable at 19.9%, ahead of Antibot (21.6%) and CAPTCHA (21.5%). Rate Limiting is lowest among secondaries at 13.8%.
Barrier Landscape
JavaScript tops Pakistan's active secondary defences
| Value | |
|---|---|
| WAF | 93.1% |
| Antibot | 21.6% |
| CAPTCHA | 21.5% |
| JavaScript | 43.5% |
| Rate Limiting | 13.8% |
| TLS | 19.9% |
| Has robots.txt | 80.3% |
| Blocks AI crawlers | 13.6% |
WAF leads at 93.1%. JavaScript is the top secondary at 43.5%. Antibot (21.6%) and CAPTCHA (21.5%) are separated by a single tenth of a percentage point. TLS stands at 19.9%, with Rate Limiting at 13.8%.
Two-barrier sites account for more than a third of domains
| % of sites | |
|---|---|
| 0 barriers | 4.4% |
| 1 barrier | 28.1% |
| 2 barriers | 35.1% |
| 3 barriers | 18.6% |
| 4 barriers | 9.6% |
| 5 barriers | 3.8% |
| 6 barriers | 0.3% |
35.1% of Pakistan sites carry exactly two barriers — the modal count. Sites with three or more barriers make up 32.3%. Just 4.4% have no barriers, the lowest zero-barrier share of these five figures.
Most Pakistan sites are Easy to access
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
The browser-based median recommended tier is 2 and the mean is 1.73. Tier 1 and Tier 2 together account for 86.8% of sites. Fewer than one in seven Pakistan sites requires more than standard browser-based retrieval.
Simple and Easy sites are almost evenly split
| % of sites | |
|---|---|
| 1 - Simple | 44.5% |
| 2 - Easy | 42.3% |
| 3 - Moderate | 10.1% |
| 4 - Complex | 1.9% |
| 5 - Advanced | 1.2% |
44.5% of sites are Tier 1 (Simple) and 42.3% are Tier 2 (Easy). Tier 3 covers 10.1%, with Tier 4 (1.9%) and Tier 5 (1.2%) making up the small residual requiring advanced extraction techniques.
robots.txt
AI crawlers are the most-cited bot category in robots.txt
| % of sites | |
|---|---|
| AI crawlers | 15.4% |
| SEO / search indexing | 12% |
| SEO audit tools | 13.8% |
| Commercial scrapers | 8.2% |
| Social / content scrapers | 6.2% |
| Archive / research crawlers | 4.6% |
| Monitoring / uptime bots | 0.5% |
15.4% of all 1,110 Pakistan domains name an AI crawler in robots.txt — the most of any category. SEO audit tools follow at 13.8% and SEO/search indexing bots at 12.0%. Monitoring bots are named by only 0.5%.
Path-mixed rules are more common than blanket approaches
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 41.9% |
| Path-level mixed rules for same agent | 33.5% |
| Allow-only (no disallow rules) | 1.9% |
| Mixed AI posture (block some, allow others) | 2.5% |
| Blanket Disallow: / | 2.8% |
41.9% of all sites use uniform per-agent rules. A substantial 33.5% apply path-level mixed rules. 19.7% have no robots.txt. Blanket Disallow: / covers 2.8% of sites, and allow-only posture just 1.9%.
GPTBot and CCBot are the most frequently blocked AI agents
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 10.9% | 2.3% |
| CCBot | 8.5% | 1.4% |
| ClaudeBot | 8.3% | 1.3% |
| Google-Extended | 8.2% | 1.8% |
| Bytespider | 6.9% | 0.4% |
| meta-externalagent | 2.9% | 0.1% |
| ChatGPT-User | 5.9% | 2.7% |
| PerplexityBot | 6% | 2.2% |
| Applebot-Extended | 4.5% | 0.3% |
| OAI-SearchBot | 4.7% | 2.3% |
| anthropic-ai | 5% | 0.9% |
10.9% of all Pakistan sites name GPTBot in a Disallow rule, the highest rate. CCBot follows at 8.5% and ClaudeBot at 8.3%. ChatGPT-User is the most explicitly allowed agent at 2.7%.
Five sectors have every site naming at least one bot to restrict
| % of sites | |
|---|---|
| Consumer Electronics | 100% |
| Design | 100% |
| Entertainment | 100% |
| Retail | 100% |
| Visual Art | 100% |
| Adult | 90% |
| Banking | 90% |
| Civil Engineering | 90% |
| Comics & Animation | 90% |
| Computer & Video Games | 90% |
| General | 90% |
| Healthcare | 90% |
| Machinery | 90% |
| Mass Media | 90% |
| Newspapers | 90% |
Consumer Electronics, Design, Entertainment, Retail, and Visual Art reach 100% — every site names at least one bot in a Disallow rule. Ten further sectors, including Banking and Healthcare, reach 90%.
Retail alone reaches the highest explicit bot Allow rate
| % of sites | |
|---|---|
| Retail | 80% |
| Photography | 70% |
| Publishing | 70% |
| Science | 70% |
| Airlines | 60% |
| Architecture | 60% |
| Automotive | 60% |
| Graphic Design | 60% |
| IT Services | 60% |
| Insurance | 60% |
| Oil & Gas | 60% |
| Real Estate | 60% |
| Software & Dev | 60% |
| Sporting Goods | 60% |
| Telecom | 60% |
80% of Retail sites name at least one bot in an explicit Allow rule — the highest in Pakistan. Photography, Publishing, and Science follow at 70%. Eleven further sectors, including Airlines and Oil & Gas, each reach 60%.
Industry Patterns
Hospitality is the hardest sector to access programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Hospitality | 90% | 30% | 0% | 60% | 30% | 40% | 60% | 10% | Tier 3 |
| Real Estate | 100% | 10% | 10% | 50% | 50% | 30% | 60% | 20% | Tier 3 |
| Equipment & Supplies | 90% | 40% | 10% | 60% | 20% | 10% | 60% | 10% | Tier 3 |
| Jewellery & Luxury | 100% | 10% | 10% | 40% | 50% | 60% | 70% | 10% | Tier 2 |
| Insurance | 80% | 30% | 10% | 40% | 10% | 20% | 80% | 0% | Tier 2 |
| Retail | 100% | 10% | 0% | 70% | 10% | 10% | 100% | 50% | Tier 2 |
| Apparel & Fashion | 100% | 0% | 10% | 50% | 20% | 40% | 70% | 20% | Tier 2 |
| Human Resources | 100% | 80% | 60% | 50% | 40% | 20% | 70% | 10% | Tier 2 |
| Medical Devices | 90% | 20% | 10% | 60% | 20% | 20% | 90% | 10% | Tier 2 |
| Warehousing | 100% | 60% | 40% | 70% | 50% | 10% | 60% | 10% | Tier 2 |
| Beauty & Cosmetics | 80% | 30% | 20% | 30% | 40% | 30% | 80% | 10% | Tier 2 |
| Computer Hardware | 90% | 20% | 0% | 30% | 20% | 20% | 70% | 10% | Tier 2 |
| Customer Services | 100% | 20% | 30% | 50% | 30% | 20% | 60% | 0% | Tier 2 |
| Restaurants | 90% | 50% | 30% | 60% | 30% | 10% | 70% | 20% | Tier 2 |
| Food & Beverages | 100% | 50% | 40% | 80% | 40% | 0% | 70% | 50% | Tier 2 |
Hospitality has the highest mean recommended tier at 2.80, followed by Real Estate at 2.70. Equipment & Supplies and Jewellery & Luxury both score 2.60 — the latter notable for 60% TLS deployment, highest in the top-tier group.
Adult sector holds the most accessible floor
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Consumer Electronics | 80% | 10% | 10% | 40% | 10% | 40% | 100% | 20% | Tier 2 |
| Crafts | 100% | 30% | 30% | 50% | 20% | 10% | 80% | 20% | Tier 2 |
| Maritime | 90% | 10% | 10% | 30% | 10% | 30% | 90% | 10% | Tier 2 |
| Oil & Gas | 100% | 20% | 0% | 70% | 0% | 30% | 80% | 10% | Tier 2 |
| Sporting Goods | 100% | 10% | 20% | 50% | 30% | 30% | 70% | 0% | Tier 2 |
| Transport & Logistics | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Wholesalers | 80% | 10% | 30% | 50% | 30% | 30% | 70% | 10% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Furniture | 100% | 50% | 20% | 60% | 20% | 40% | 70% | 10% | Tier 2 |
| Mail & Delivery | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Public Safety | 100% | 40% | 40% | 50% | 20% | 30% | 90% | 10% | Tier 2 |
| Recruitment & Staffing | 100% | 50% | 60% | 30% | 10% | 30% | 70% | 20% | Tier 2 |
| Renewable Energy | 100% | 10% | 10% | 70% | 10% | 30% | 90% | 0% | Tier 2 |
| Wine & Spirits | 100% | 50% | 10% | 50% | 30% | 0% | 80% | 40% | Tier 2 |
| Adult | 100% | 40% | 30% | 80% | 10% | 20% | 90% | 10% | Tier 2 |
Among more accessible sectors, Adult has the lowest mean tier at 1.90 — despite 100% WAF and 80% JavaScript deployment. Fourteen sectors above it share means of 2.00 to 2.10, with Recruitment & Staffing carrying 60% CAPTCHA at that level.