Overview
JavaScript leads all secondary barriers across Australian sites
| Value | |
|---|---|
| WAF | 92.9% |
| Antibot | 21.6% |
| CAPTCHA | 24.4% |
| JavaScript | 41.9% |
| Rate Limiting | 13.5% |
| TLS | 17.6% |
WAF adoption stands at 92.9% of Australia's 1,111 surveyed sites. JavaScript leads secondary barriers at 41.9%, followed by CAPTCHA at 24.4% and Antibot at 21.6%.
Barrier Landscape
JavaScript is the leading secondary barrier in Australia
| Value | |
|---|---|
| WAF | 92.9% |
| Antibot | 21.6% |
| CAPTCHA | 24.4% |
| JavaScript | 41.9% |
| Rate Limiting | 13.5% |
| TLS | 17.6% |
| Has robots.txt | 79.8% |
| Blocks AI crawlers | 14.5% |
WAF at 92.9% is the dominant barrier across Australia's surveyed sites. JavaScript (41.9%) is the most-deployed secondary barrier; CAPTCHA (24.4%) and Antibot (21.6%) follow.
Two-barrier sites are the most common configuration
| % of sites | |
|---|---|
| 0 barriers | 4.8% |
| 1 barrier | 29.3% |
| 2 barriers | 34.5% |
| 3 barriers | 17.8% |
| 4 barriers | 8.6% |
| 5 barriers | 4.7% |
| 6 barriers | 0.4% |
4.8% of Australia sites present no detectable barriers. The modal outcome is two barriers, covering 34.5% of sites. Sites with three or more barriers account for 31.5%.
Most Australian sites are Easy to access programmatically
| Mean recommended tier | |
|---|---|
| All sites | 1.7 |
A mean of 1.70 and median of 2 characterise Australia's access difficulty. Tier 1 accounts for 46.4% of sites; the 13.2% at Tier 3–5 pull the mean upward.
Nearly nine in 10 sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 46.4% |
| 2 - Easy | 40.3% |
| 3 - Moderate | 10.5% |
| 4 - Complex | 1.8% |
| 5 - Advanced | 0.9% |
Tier 1 covers 46.4% of Australia's sites; Tier 2 adds 40.3%, bringing the combined Tier 1+2 share to 86.7%. Tier 3 accounts for 10.5%, Tier 4 for 1.8%, Tier 5 for 0.9%.
robots.txt
AI crawlers are the most-named bot category in robots.txt
| % of sites | |
|---|---|
| AI crawlers | 16% |
| SEO / search indexing | 11.8% |
| SEO audit tools | 13.9% |
| Commercial scrapers | 8.5% |
| Social / content scrapers | 5.9% |
| Archive / research crawlers | 4.8% |
| Monitoring / uptime bots | 0.5% |
AI crawlers are mentioned by 16.0% of Australia's 1,111 surveyed sites — the most of any bot category. SEO / search indexing follow at 11.8%. Figures are percentages of all surveyed sites, including those without robots.txt.
Uniform per-agent rules are the most common posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 41.5% |
| Path-level mixed rules for same agent | 32.8% |
| Allow-only (no disallow rules) | 2.6% |
| Mixed AI posture (block some, allow others) | 2.6% |
| Blanket Disallow: / | 2.8% |
20.2% of Australia's sites have no robots.txt. Uniform rules is the most common posture at 41.5% of all sites, followed by Path-mixed rules at 32.8%. Blanket Disallow: / covers 2.8%.
GPTBot is the most blocked agent in Australia
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 12% | 2.7% |
| CCBot | 9.5% | 1.1% |
| ClaudeBot | 8.6% | 1.4% |
| Google-Extended | 8.6% | 1.7% |
| Bytespider | 7.6% | 0.5% |
| meta-externalagent | 3.2% | — |
| ChatGPT-User | 6.2% | 2.5% |
| PerplexityBot | 6.1% | 2.2% |
| Applebot-Extended | 5.4% | 0.5% |
| OAI-SearchBot | 4.8% | 2.1% |
| anthropic-ai | 5.5% | 0.8% |
Of Australia's 1,111 surveyed sites, GPTBot is Disallowed by 12.0% and explicitly Allowed by 2.7%. GPTBot has the highest explicit Allow rate at 2.7%. All values include sites without robots.txt.
Civil engineering sites all name at least one bot to restrict
| % of sites | |
|---|---|
| Civil Engineering | 100% |
| Computer & Video Games | 100% |
| Design | 100% |
| Entertainment | 100% |
| Newspapers | 100% |
| Retail | 100% |
| Visual Art | 100% |
| Adult | 90% |
| Agriculture | 90% |
| Automotive | 90% |
| Comics & Animation | 90% |
| Consumer Electronics | 90% |
| Events | 90% |
| Food & Beverages | 90% |
| General | 90% |
Seven sectors each show a 100% named-bot Disallow rate — every site in those sectors names at least one bot to restrict in robots.txt. The next sectors in the ranking stand at 90%.
Automotive leads sectors on explicit bot Allow rules
| % of sites | |
|---|---|
| Automotive | 90% |
| Airlines | 70% |
| Civil Engineering | 70% |
| Events | 70% |
| Photography | 70% |
| Publishing | 70% |
| Real Estate | 70% |
| Retail | 70% |
| Sporting Goods | 70% |
| Telecom | 70% |
| Farms & Ranches | 60% |
| Graphic Design | 60% |
| Non-Profit | 60% |
| Outsourcing | 60% |
| Restaurants | 60% |
Automotive has the highest named-bot Allow rate in Australia at 90% of sites in that sector. Airlines follows at 70%.
Industry Patterns
Hospitality is the hardest sector to access programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Hospitality | 90% | 30% | 0% | 60% | 30% | 40% | 60% | 10% | Tier 3 |
| Jewellery & Luxury | 100% | 30% | 20% | 30% | 40% | 50% | 70% | 0% | Tier 3 |
| Equipment & Supplies | 90% | 30% | 20% | 70% | 40% | 10% | 50% | 10% | Tier 3 |
| Real Estate | 90% | 30% | 30% | 50% | 50% | 10% | 80% | 20% | Tier 2 |
| Automotive | 100% | 10% | 0% | 40% | 40% | 20% | 90% | 40% | Tier 2 |
| Beauty & Cosmetics | 90% | 40% | 30% | 40% | 40% | 0% | 70% | 10% | Tier 2 |
| Computer Hardware | 90% | 20% | 0% | 30% | 20% | 20% | 70% | 10% | Tier 2 |
| Human Resources | 100% | 70% | 60% | 40% | 40% | 20% | 70% | 10% | Tier 2 |
| Retail | 100% | 0% | 0% | 60% | 20% | 20% | 100% | 50% | Tier 2 |
| Warehousing | 100% | 50% | 50% | 60% | 30% | 10% | 60% | 0% | Tier 2 |
| Apparel & Fashion | 90% | 10% | 30% | 60% | 20% | 20% | 80% | 10% | Tier 2 |
| Construction | 90% | 20% | 20% | 50% | 30% | 0% | 60% | 10% | Tier 2 |
| Furniture | 100% | 30% | 40% | 40% | 20% | 20% | 50% | 10% | Tier 2 |
| Mail & Delivery | 70% | 40% | 10% | 20% | 0% | 20% | 80% | 0% | Tier 2 |
| Consumer Electronics | 80% | 10% | 20% | 40% | 10% | 30% | 90% | 10% | Tier 2 |
Hospitality averages mean tier 2.80 across Australia's surveyed sites — the highest of any industry. Jewellery & Luxury follows at 2.80. Industries ranked by mean recommended browser tier.
Six sectors share Australia's most accessible mean tier
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Maritime | 90% | 10% | 10% | 40% | 10% | 30% | 90% | 10% | Tier 2 |
| Food & Beverages | 100% | 60% | 30% | 60% | 40% | 0% | 90% | 40% | Tier 2 |
| Banking | 100% | 20% | 10% | 40% | 10% | 10% | 60% | 0% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Medical Devices | 100% | 0% | 20% | 50% | 10% | 0% | 100% | 0% | Tier 2 |
| Public Safety | 100% | 20% | 40% | 50% | 10% | 30% | 90% | 10% | Tier 2 |
| Restaurants | 90% | 30% | 30% | 60% | 40% | 20% | 90% | 20% | Tier 2 |
| Textiles & Nonwovens | 100% | 10% | 30% | 50% | 10% | 20% | 90% | 10% | Tier 2 |
| Transport & Logistics | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Education | 100% | 20% | 10% | 30% | 10% | 10% | 70% | 10% | Tier 2 |
| Events | 100% | 10% | 10% | 70% | 10% | 0% | 90% | 10% | Tier 2 |
| Manufacturing | 80% | 10% | 10% | 30% | 0% | 40% | 70% | 0% | Tier 2 |
| Recruitment & Staffing | 100% | 50% | 60% | 30% | 10% | 30% | 70% | 20% | Tier 2 |
| Reference | 90% | 60% | 30% | 50% | 30% | 10% | 60% | 30% | Tier 2 |
| Sporting Goods | 80% | 10% | 40% | 20% | 10% | 30% | 80% | 20% | Tier 2 |
Six sectors cluster at Australia's most accessible mean tier of 1.90. Industries ranked by mean recommended browser tier.