Overview
Restaurants carry heavy JavaScript and Rate Limiting but skip robots.txt
| Value | |
|---|---|
| WAF | 91% |
| Antibot | 28% |
| CAPTCHA | 19% |
| JavaScript | 57% |
| Rate Limiting | 41% |
| TLS | 15% |
JavaScript (57%) and Rate Limiting (41%) are the standout barriers in Restaurants — both well above the global norm. WAF (91%) and antibot (28%) are elevated. Despite this technical stack, only half of sites publish any robots.txt.
Barrier Landscape
JavaScript reaches 57% in Restaurants — the highest in the Travel & Hospitality cohort
| Value | |
|---|---|
| WAF | 91% |
| Antibot | 28% |
| CAPTCHA | 19% |
| JavaScript | 57% |
| Rate Limiting | 41% |
| TLS | 15% |
| Has robots.txt | 50% |
| Blocks AI crawlers | 5% |
WAF leads at 91%. JavaScript (57%) and Rate Limiting (41%) are both elevated. Antibot (28%) and CAPTCHA (19%) round out a profile heavier on active barriers than most sectors. TLS (15%) sits near the norm.
Restaurants commonly stack three or more access barriers per site
| % of sites | |
|---|---|
| 0 barriers | 4% |
| 1 barrier | 23% |
| 2 barriers | 23% |
| 3 barriers | 30% |
| 4 barriers | 9% |
| 5 barriers | 10% |
| 6 barriers | 1% |
Three-barrier stacks are the modal count at 30%; 1% carry all six. Combined, 50% of Restaurants carry three or more barriers — higher than most sectors surveyed. Only 4% carry none.
Restaurants are predominantly Easy to access despite their barrier load
| Mean recommended tier | |
|---|---|
| All sites | 1.98 |
A mean recommended tier of 1.98 and a median of Easy place Restaurants around the middle of the sectors surveyed. 27% of sites are Simple to access; 20% require Moderate or above.
Easy tier covers more than half of Restaurant sites
| % of sites | |
|---|---|
| 1 - Simple | 27% |
| 2 - Easy | 53% |
| 3 - Moderate | 16% |
| 4 - Complex | 3% |
| 5 - Advanced | 1% |
Easy (53%) is the most common tier. Simple and Easy together account for 80% of sites, while 16% land at Moderate and 4% at Complex or Advanced.
robots.txt
Half of Restaurant sites publish no robots.txt at all
| % of sites | |
|---|---|
| Publish a valid robots.txt | 50% |
Only 50% of Restaurants have a valid robots.txt — one of the lowest rates across all surveyed sectors. Half the sector leaves crawl policy entirely undefined by any explicit file.
Most Restaurant sites name no bots at all in robots.txt
| % of sites | |
|---|---|
| 0 | 53% |
| 1 | 33% |
| 2–4 | 8% |
| 5–9 | 2% |
| 10–15 | 0% |
| 16+ | 4% |
53% of Restaurant sites name zero bot agents in their robots.txt; 33% name exactly one. The mean of 1.8 named agents is among the lowest of any sector, reflecting the sector's low engagement with named crawler policy.
All bot categories see low attention in Restaurant robots.txt files
| % of sites | |
|---|---|
| AI crawlers | 6% |
| SEO / search indexing | 5% |
| SEO audit tools | 5% |
| Commercial scrapers | 4% |
| Social / content scrapers | 4% |
| Archive / research crawlers | 2% |
| Monitoring / uptime bots | 2% |
No bot category exceeds 6% mention rate across Restaurant sites. AI crawlers lead at 6%, ahead of SEO audit tools and SEO / search indexing bots, both at 5%. Monitoring bots and archive crawlers each appear at 2%.
Uniform per-agent rules dominate the half of Restaurants that use robots.txt
| % of sites | |
|---|---|
| Uniform per-agent rules | 27% |
| Path-level mixed rules | 17% |
| Allow-only (no Disallow rules) | 2% |
| Mixed AI posture | 2% |
| Blanket Disallow: / | 4% |
Of the 50% of Restaurant sites with robots.txt, uniform per-agent rules lead at 27%, with path-level mixed rules at 17%. A 4% blanket Disallow rate is higher than most sectors. Just 2% maintain a mixed AI posture.
Restaurant robots.txt targets the wildcard and common search bots
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 44% | 20% |
| Googlebot | 4% | 3% |
| Twitterbot | 3% | 4% |
| AhrefsBot | 3% | 3% |
| Applebot | 3% | 3% |
| DuckDuckBot | 3% | 3% |
| Googlebot-Mobile | 3% | 3% |
| OAI-SearchBot | 3% | 3% |
| Amazonbot | 3% | 2% |
| ChatGPT-User | 3% | 2% |
| PerplexityBot | 2% | 3% |
| AdsBot-Google | 1% | 4% |
| AdIdxBot | 2% | 2% |
| BingPreview | 2% | 2% |
| Bingbot | 2% | 2% |
| Claude-SearchBot | 2% | 2% |
| Claude-User | 2% | 2% |
| FacebookExternalHit | 2% | 2% |
| Facebot | 2% | 2% |
| Google-InspectionTool | 2% | 2% |
| Google-Read-Aloud | 2% | 2% |
| GoogleOther | 2% | 2% |
| Googlebot-Image | 2% | 2% |
| Googlebot-Video | 2% | 2% |
| LinkedInBot | 2% | 2% |
Values are percentages of all 100 Restaurant sites surveyed. The wildcard (*) leads at 44% Disallow. Named rules focus on Googlebot, Twitterbot, AhrefsBot and Applebot. AI-specific crawlers rarely appear as named targets.
Industry Cohort
Restaurants lead the Travel & Hospitality cohort on JavaScript and Rate Limiting
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Airlines | 76% | 24% | 11% | 51% | 27% | 22% | 69% | 9% | Tier 2 |
| Events Services | 83% | 17% | 21% | 41% | 26% | 12% | 65% | 3% | Tier 1 |
| Hospitality | 86% | 18% | 17% | 47% | 27% | 19% | 76% | 9% | Tier 2 |
| Recreational Facilities | 95% | 23% | 21% | 51% | 21% | 11% | 71% | 6% | Tier 1 |
| Restaurants | 91% | 28% | 19% | 57% | 41% | 15% | 50% | 5% | Tier 2 |
| Travel & Tourism | 81% | 27% | 8% | 55% | 31% | 26% | 78% | 21% | Tier 2 |
Restaurants' 57% JavaScript adoption and 41% Rate Limiting rate lead the Travel & Hospitality cohort. robots.txt coverage (50%) is the lowest in the group. Despite active barriers, tier difficulty (mean 1.88) sits mid-cohort.