Overview
Airlines invest heavily in JavaScript and Rate Limiting — WAF is a weak point
| Value | |
|---|---|
| WAF | 76% |
| Antibot | 24% |
| CAPTCHA | 11% |
| JavaScript | 51% |
| Rate Limiting | 27% |
| TLS | 22% |
At 76%, Airlines WAF coverage is the lowest in this batch and well below the global norm. JavaScript (51%), Rate Limiting (27%) and TLS (22%) are all elevated, reflecting a sector that layers active and network barriers rather than relying on WAF alone.
Barrier Landscape
JavaScript and Rate Limiting define the Airlines barrier profile
| Value | |
|---|---|
| WAF | 76% |
| Antibot | 24% |
| CAPTCHA | 11% |
| JavaScript | 51% |
| Rate Limiting | 27% |
| TLS | 22% |
| Has robots.txt | 69% |
| Blocks AI crawlers | 9% |
WAF leads at 76% — low for the sector. JavaScript (51%) is the dominant secondary barrier, followed by Rate Limiting (27%) and TLS (22%). Antibot (24%) and CAPTCHA (11%) round out a profile that leans on active challenges.
Airlines stack more barriers per site than most sectors
| % of sites | |
|---|---|
| 0 barriers | 12% |
| 1 barrier | 25% |
| 2 barriers | 22% |
| 3 barriers | 26% |
| 4 barriers | 11% |
| 5 barriers | 4% |
| 6 barriers | 0% |
3+ barrier stacks account for 41% of Airlines sites — higher than the global norm. 12% carry none, while the modal count is 3 (26%). Airlines show a wide spread from open to heavily fortified.
Airlines are among the harder sectors to access programmatically
| Mean recommended tier | |
|---|---|
| All sites | 2.02 |
A mean recommended tier of 2.02 and a median of Easy place Airlines around the middle of the sectors surveyed. 32% of sites are Simple to access; 24% require Moderate or above.
Easy tier dominates Airlines, with a significant Moderate tail
| % of sites | |
|---|---|
| 1 - Simple | 31.6% |
| 2 - Easy | 44.9% |
| 3 - Moderate | 16.3% |
| 4 - Complex | 4.1% |
| 5 - Advanced | 3.1% |
Easy (45%) is the most common tier. Simple and Easy together account for 76% of sites, while 16% land at Moderate and 7% at Complex or Advanced.
robots.txt
Nearly a third of Airlines publish no robots.txt
| % of sites | |
|---|---|
| Publish a valid robots.txt | 69% |
69% of Airlines sites have a valid robots.txt. The 31% with no file include some of the sector's largest carriers, leaving a substantial share of crawl policy undefined.
Airlines robots.txt files are sparse — most name one agent or none
| % of sites | |
|---|---|
| 0 | 36% |
| 1 | 31% |
| 2–4 | 9% |
| 5–9 | 11% |
| 10–15 | 11% |
| 16+ | 2% |
36% of Airlines sites name no agents in robots.txt; 31% name exactly one. A bimodal tail includes sites naming 12 agents (7%) or 13 agents (4%). The overall mean of 3.2 reflects low typical engagement with named bot policy.
SEO audit tools lead Airlines robots.txt mentions ahead of AI crawlers
| % of sites | |
|---|---|
| AI crawlers | 13% |
| SEO / search indexing | 9% |
| SEO audit tools | 14% |
| Commercial scrapers | 3% |
| Social / content scrapers | 3% |
| Archive / research crawlers | 2% |
| Monitoring / uptime bots | 0% |
SEO audit tools are mentioned by 14% of Airlines sites — the highest category — followed by AI crawlers at 13%. No other bot category exceeds 9%. Airlines show more concern with SEO scraping than AI training.
Path-level mixed rules dominate Airlines robots.txt posture
| % of sites | |
|---|---|
| Uniform per-agent rules | 22% |
| Path-level mixed rules | 43% |
| Allow-only (no Disallow rules) | 2% |
| Mixed AI posture | 6% |
| Blanket Disallow: / | 2% |
Of the 69% of Airlines with a robots.txt, path-level mixed rules lead at 43%, with uniform per-agent rules at 22%. A 6% mixed AI posture — blocking some crawlers while allowing others — is higher than most sectors in this batch.
Airlines robots.txt focuses on Google ad bots and SEO crawlers, not AI
| Disallowed | Allowed (whitelisted) | |
|---|---|---|
| * | 61% | 38% |
| Mediapartners-Google | 13% | 7% |
| AdsBot-Google | 9% | 10% |
| AhrefsBot | 10% | 7% |
| AdsBot-Google-Mobile | 6% | 7% |
| ChatGPT-User | 3% | 8% |
| Googlebot | 6% | 5% |
| PerplexityBot | 3% | 8% |
| GPTBot | 5% | 5% |
| NPBot | 8% | — |
| AdIdxBot | 3% | 5% |
| google-hoteladsverifier | 3% | 4% |
| CCBot | 4% | 3% |
| HenryTheMiragoRobot | 7% | — |
| Leikibot | 7% | — |
| Meta-ExternalAgent | 7% | — |
| ShopWiki | 7% | — |
| YisouSpider | 7% | — |
| appie | 7% | — |
| gigabot | 7% | — |
| psbot | 7% | — |
| OAI-SearchBot | 1% | 6% |
| Claude-User | 1% | 5% |
| Google-Extended | 2% | 2% |
| Google-HotelAdsVerifier | 2% | 2% |
Values are percentages of all 100 Airlines sites surveyed. The wildcard (*) leads at 61% Disallow. Named rules target Google's ad crawlers (Mediapartners-Google, AdsBot-Google) and AhrefsBot ahead of any AI-specific agent.
Industry Cohort
Airlines are the hardest sub-category to access in Travel & Hospitality
| WAF | Antibot | CAPTCHA | JavaScript | Rate Limit | TLS | robots.txt | Blocks AI | Tier | |
|---|---|---|---|---|---|---|---|---|---|
| Airlines | 76% | 24% | 11% | 51% | 27% | 22% | 69% | 9% | Tier 2 |
| Events Services | 83% | 17% | 21% | 41% | 26% | 12% | 65% | 3% | Tier 1 |
| Hospitality | 86% | 18% | 17% | 47% | 27% | 19% | 76% | 9% | Tier 2 |
| Recreational Facilities | 95% | 23% | 21% | 51% | 21% | 11% | 71% | 6% | Tier 1 |
| Restaurants | 91% | 28% | 19% | 57% | 41% | 15% | 50% | 5% | Tier 2 |
| Travel & Tourism | 81% | 27% | 8% | 55% | 31% | 26% | 78% | 21% | Tier 2 |
Airlines' mean tier of 1.97 is the highest in the Travel & Hospitality cohort. JavaScript (51%) and Rate Limiting (27%) also lead the group. WAF (76%) is notably the cohort's lowest — reflecting a sector that trades firewall coverage for active challenge layers.