Overview
German sites carry barriers typical of global norms
| Value | |
|---|---|
| WAF | 93.5% |
| Antibot | 21.2% |
| CAPTCHA | 22.8% |
| JavaScript | 41.1% |
| Rate Limiting | 12.2% |
| TLS | 18.5% |
Every German barrier rate sits within one percentage point of the global benchmark — WAF 93.5% vs global 93.1%, JavaScript 41.1% vs 41.3%, and so on across all six. No barrier type pulls the radar shape measurably away from the European norm.
Barrier Landscape
JavaScript is Germany's most common secondary barrier
| Value | |
|---|---|
| WAF | 93.5% |
| Antibot | 21.2% |
| CAPTCHA | 22.8% |
| JavaScript | 41.1% |
| Rate Limiting | 12.2% |
| TLS | 18.5% |
| Has robots.txt | 82% |
| Blocks AI crawlers | 14.1% |
WAF at 93.5% is in line with the global 93.1%. JavaScript (41.1%) is the secondary barrier that most differentiates German sites from a plain HTTP scrape. CAPTCHA (22.8%) and Antibot (21.2%) are closely matched at the next tier down.
Most German sites carry at least two barriers
| % of sites | |
|---|---|
| 0 barriers | 4.2% |
| 1 barrier | 30.1% |
| 2 barriers | 35.6% |
| 3 barriers | 17.2% |
| 4 barriers | 8.6% |
| 5 barriers | 4.1% |
| 6 barriers | 0.3% |
Only 4.2% of sites present no detectable barriers. The modal outcome is two barriers (35.6%), most commonly WAF combined with JavaScript. Sites at three or more account for 30%, with four-plus reaching 13%.
German sites sit comfortably in the Easy access band
| Mean recommended tier | |
|---|---|
| All sites | 1.6 |
A mean of 1.65 and a median of 2 place Germany just above the Simple/Easy boundary. Tier 1 covers 49.1% of sites, pulling the mean down; the 10.9% requiring Tier 3–5 push it back up.
Nearly all German sites are Simple or Easy to access
| % of sites | |
|---|---|
| 1 - Simple | 49.1% |
| 2 - Easy | 40% |
| 3 - Moderate | 8.6% |
| 4 - Complex | 1.4% |
| 5 - Advanced | 0.9% |
Tier 1 (49.1%) and Tier 2 (40.0%) together account for 89.1% of the sample. The 8.6% at Tier 3 reflects concentrated difficulty in accommodation, luxury, and property sectors. Tier 4 (1.4%) and Tier 5 (0.9%) are small but present.
robots.txt
AI crawlers top Germany's robots.txt bot categories
| % of sites | |
|---|---|
| AI crawlers | 15.9% |
| SEO / search indexing | 13.1% |
| SEO audit tools | 14.6% |
| Commercial scrapers | 8.5% |
| Social / content scrapers | 6.1% |
| Archive / research crawlers | 5.2% |
| Monitoring / uptime bots | 0.5% |
AI crawlers (15.9%) and SEO audit tools (14.6%) lead by a narrow margin. All figures are percentages of all 1,111 German sites including those without robots.txt. Monitoring bots (0.5%) are effectively absent.
Uniform rules are Germany's most common robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules (no path mixing) | 44.4% |
| Path-level mixed rules for same agent | 32.9% |
| Allow-only (no disallow rules) | 2.3% |
| Mixed AI posture (block some, allow others) | 2.2% |
| Blanket Disallow: / | 2.4% |
18.0% of sites publish no robots.txt. Uniform posture (44.4%) leads, ahead of path-mixed (32.9%). Blanket Disallow: / stands at 2.4% and Mixed-AI at 2.2%.
GPTBot faces the most Disallow rules in Germany's robots.txt
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| GPTBot | 11.5% | 2.3% |
| CCBot | 9% | 1.1% |
| ClaudeBot | 8.6% | 1.3% |
| Google-Extended | 8.3% | 1.2% |
| Bytespider | 7.4% | 0.4% |
| meta-externalagent | 2.8% | — |
| ChatGPT-User | 5.9% | 2.4% |
| PerplexityBot | 6.1% | 1.9% |
| Applebot-Extended | 5.2% | 0.5% |
| OAI-SearchBot | 4.1% | 1.9% |
| anthropic-ai | 5.3% | 0.6% |
GPTBot (11.5%) leads the block side by more than 2 points. On the allow side, ChatGPT-User (2.4%) and GPTBot (2.3%) are nearly level. All values are percentages of all 1,111 German sites.
Seven German sectors all name at least one bot to restrict
| % of sites | |
|---|---|
| Computer & Video Games | 100% |
| Design | 100% |
| Entertainment | 100% |
| Plastics & Polymers | 100% |
| Retail | 100% |
| Veterinary | 100% |
| Visual Art | 100% |
| Adult | 90% |
| Agriculture | 90% |
| Apparel & Fashion | 90% |
| Automotive | 90% |
| Beauty & Cosmetics | 90% |
| Comics & Animation | 90% |
| Crafts | 90% |
| Food & Beverages | 90% |
Seven sectors reach 100%: Computer & Video Games, Design, Entertainment, Plastics & Polymers, Retail, Veterinary, and Visual Art — every sampled site in each names at least one crawler in a Disallow rule. The remaining top-15 sectors sit at 90%.
Germany's Automotive and Food & Beverages sectors lead in allowing crawlers
| % of sites | |
|---|---|
| Automotive | 80% |
| Food & Beverages | 80% |
| Science | 80% |
| Airlines | 70% |
| Architecture | 70% |
| Crafts | 70% |
| Farms & Ranches | 70% |
| Photography | 70% |
| Publishing | 70% |
| Travel & Tourism | 70% |
| Graphic Design | 60% |
| Investment | 60% |
| Public Relations | 60% |
| Real Estate | 60% |
| Retail | 60% |
Automotive, Food & Beverages, and Science all reach 80%. Automotive also appears in the Disallow chart — some sites in the sector simultaneously name bots to restrict while others explicitly allow them. Real Estate trails at 60%.
Industry Patterns
Hospitality is Germany's hardest sector to access programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Hospitality | 90% | 20% | 0% | 50% | 40% | 50% | 60% | 20% | Tier 3 |
| Real Estate | 100% | 10% | 10% | 40% | 30% | 20% | 60% | 20% | Tier 3 |
| Jewellery & Luxury | 100% | 20% | 20% | 30% | 40% | 60% | 60% | 0% | Tier 3 |
| Computer Hardware | 100% | 20% | 0% | 30% | 30% | 20% | 80% | 10% | Tier 2 |
| Automotive | 100% | 10% | 10% | 40% | 30% | 30% | 90% | 20% | Tier 2 |
| Beauty & Cosmetics | 90% | 40% | 20% | 40% | 10% | 20% | 90% | 30% | Tier 2 |
| Apparel & Fashion | 90% | 10% | 20% | 50% | 20% | 40% | 90% | 10% | Tier 2 |
| Furniture | 100% | 40% | 30% | 40% | 20% | 30% | 70% | 20% | Tier 2 |
| Human Resources | 100% | 70% | 60% | 50% | 40% | 20% | 80% | 10% | Tier 2 |
| Restaurants | 90% | 40% | 20% | 50% | 30% | 20% | 60% | 10% | Tier 2 |
| Retail | 90% | 10% | 0% | 50% | 20% | 30% | 100% | 40% | Tier 2 |
| Telecom | 90% | 0% | 0% | 60% | 0% | 10% | 90% | 10% | Tier 2 |
| Consumer Electronics | 80% | 20% | 10% | 40% | 20% | 40% | 80% | 10% | Tier 2 |
| Customer Services | 100% | 20% | 30% | 60% | 30% | 10% | 60% | 0% | Tier 2 |
| Equipment & Supplies | 90% | 30% | 20% | 30% | 10% | 10% | 70% | 10% | Tier 2 |
Hospitality (2.90), Real Estate (2.80), and Jewellery & Luxury (2.70) form the top three. A mid-table cluster — Apparel, Furniture, Human Resources, Restaurants, and Retail — all share a mean of 2.20, making the lower portion of the ranking flat.
Food & Beverages is Germany's most accessible sector to access programmatically
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Medical Devices | 100% | 0% | 20% | 60% | 0% | 0% | 100% | 10% | Tier 2 |
| Design | 100% | 20% | 30% | 40% | 20% | 30% | 100% | 30% | Tier 2 |
| Machinery | 90% | 0% | 10% | 50% | 0% | 0% | 90% | 10% | Tier 2 |
| Mail & Delivery | 70% | 40% | 10% | 10% | 0% | 20% | 80% | 0% | Tier 2 |
| Public Safety | 100% | 40% | 40% | 50% | 20% | 30% | 90% | 10% | Tier 2 |
| Transport & Logistics | 70% | 50% | 10% | 20% | 0% | 20% | 80% | 0% | Tier 2 |
| Adult | 100% | 40% | 30% | 80% | 10% | 20% | 90% | 10% | Tier 2 |
| Crafts | 100% | 30% | 30% | 40% | 10% | 10% | 90% | 30% | Tier 2 |
| Manufacturing | 80% | 20% | 20% | 40% | 0% | 40% | 80% | 10% | Tier 2 |
| Recruitment & Staffing | 100% | 50% | 60% | 40% | 10% | 20% | 70% | 10% | Tier 2 |
| Reference | 90% | 60% | 30% | 50% | 30% | 10% | 60% | 30% | Tier 2 |
| Sporting Goods | 80% | 20% | 30% | 10% | 40% | 30% | 70% | 0% | Tier 2 |
| Visual Art | 100% | 30% | 20% | 30% | 0% | 20% | 100% | 20% | Tier 2 |
| Warehousing | 100% | 60% | 20% | 80% | 50% | 10% | 60% | 10% | Tier 2 |
| Food & Beverages | 100% | 50% | 30% | 50% | 30% | 0% | 90% | 40% | Tier 2 |
Food & Beverages (1.85) is the floor. Eight sectors cluster at a mean tier of 1.90 and five sectors share 2.00, compressing most of the bottom 15 into a narrow band above the floor.