Line
WAF adoption climbs every single year
Sharp corners make year-on-year direction impossible to misread. No interpolation — no hidden flat spells.
WAF and JS are pulling away from the rest
WAF deployment has more than doubled since 2020. JS-rendering barriers have grown faster in percentage terms. CAPTCHAs — the old default — are quietly declining.
The barrier curve bends upward — no sign of slowing
Bezier interpolation emphasises the gentle acceleration in the middle years. The curve implies a trajectory, not just a sequence of measurements.
Block rate jumps at each algorithm update, then holds
Step rendering makes discrete policy changes legible. The flat horizontal bands confirm block rates are stable between deployments — then jolt upward.
Area
The open web keeps shrinking — year after year
The filled region gives volume to the barrier footprint. From 58% in 2020 to 81.5% in 2025, the growing mass of gated pages is impossible to dismiss.
WAF has consumed most of the stack's growth since 2022
Stacked fills reveal composition as well as magnitude. CAPTCHAs are losing ground in relative terms while WAF expands fastest — even as the total stack grows.
Bar — vertical
Finance leads — every other sector is catching up
Finance has defended at 90%+ for three consecutive years. Travel crossed 80% in 2024. Healthcare and News are the last holdouts — but both are accelerating.
WAF leads JS in every sector — but the gap is narrowing
In Finance the gap is 18 points; in News just 15. As single-page applications proliferate, JS-rendering barriers are fast becoming as common as WAFs.
Most defended sectors stack all three barrier types
Stacking reveals cumulative exposure, not just the dominant type. Finance doesn't just deploy more WAFs — it also leads on JS and secondary barriers combined.
WAF is the majority barrier type in every single sector
Normalising to 100% strips out volume and isolates composition. News is the only sector where CAPTCHA accounts for more than a fifth of the barrier mix.
Bar — horizontal
WAF blocks more pages than any other barrier type
WAF has a 21-point lead over the next nearest barrier. Bot-score detection and IP blocking, while growing, still affect fewer than one in three pages.
Barriers grew fastest where they were already highest
WAF and JS dependency both posted 8-point YoY gains. Login walls grew slower in absolute terms but faster in relative terms — up 13% from their 2024 base.
Finance layers every barrier type — News relies heavily on JS
Finance doesn't choose between barrier types — it deploys all three at scale. News sites lean disproportionately on JS rendering, consistent with their heavy SPA adoption.
Furniture and real estate defend hardest — education and consulting barely bother
The 34-point spread between the most and least defended industries dwarfs any regional difference. Your target industry is a better predictor of access friction than the country you're scraping from.
Antibot sites are far more likely to deploy every other barrier type
The coloured ghost band anchors each pair to its barrier category before a number is read. Use this pattern when the categories themselves carry meaning (canonical colour) and the two cohorts are structurally paired rather than independently ranked.
Split bar
Open access is now in the minority
Only 18.5% of popular landing pages operate without any detectable barrier. The frictionless web now belongs to a minority of informational and public-sector sites.
The vast majority of popular sites remain affordable to scrape
78% of landing pages are Zyte tier 1 — the simplest, cheapest access configuration. Only 6% require more sophisticated infrastructure, clustered in real estate, travel, and entertainment.
Circular
Cloudflare controls more than half the WAF market
The top-four providers account for 92% of detected WAF deployments. The remaining 8% is a long tail of niche and self-hosted solutions.
WAF and JS together account for two-thirds of all barriers
The barrier mix has consolidated. In 2020, six types each held a meaningful share. By 2025, WAF and JS dominate — the rest are footnotes in aggregate.
Finance and Travel are clear outliers in sector exposure
Polar area encodes magnitude through radius, not arc. Finance and Travel's wedges reach furthest — visualising not just that they're defended, but by how much.
Radar
Finance deploys every barrier type at scale
The near-circular profile means no gap in defences — a rare posture that reflects both regulatory pressure and high breach costs in financial services.
Finance and News couldn't have more different defence profiles
Three sectors overlaid: Finance's complete perimeter versus News's thin outer ring. E-commerce sits between — strong on WAF and JavaScript but underweighted on Antibot.
Scatter & bubble
More barriers mean slower pages — the correlation is hard to escape
Each point is a sampled page. The upward drift from left to right is not coincidental — each additional barrier layer adds between 0.4s and 0.9s to median TTFB.
High-traffic sites defend hardest — and pay the largest latency premium
Bubble radius encodes barrier count; x-axis shows traffic rank. The largest bubbles cluster left — the most-visited sites are also the most aggressively defended, and the slowest.
Mixed & combo
As barrier rates rise, page-load times rise in lockstep
Bars track the expanding barrier footprint (left axis); the line tracks median TTFB in seconds (right axis). Both move in the same direction across every year — defence is not free.
WAF deployments doubled — and latency overhead doubled with them
Two incompatible units — deployment count (left) and milliseconds (right) — share the same time axis. The parallel growth lines confirm the performance cost of WAF inspection scales with adoption.
Annotation
Barrier rate crossed the critical threshold in 2024 — and kept climbing
Annotations mark real context: a dashed reference line shows the 75% alert threshold; the red zone above it highlights critical territory; vertical markers flag the two regulatory events that accelerated adoption.
Finance and Travel are the only sectors above the global average
A single dashed benchmark line cuts through the bars and immediately separates leaders from laggards — without needing a callout or annotation text.
Two algorithm updates explain every major step-change in block rate
Point annotations mark the exact quarters where Cloudflare's v2 and v3 detection algorithms shipped. The step-changes align exactly — the data confirms the mechanism.
Treemap
Cloudflare's footprint dwarfs every other WAF vendor combined
Area encodes share directly — no axis, no scale, just proportional space. Cloudflare's 58% block occupies more than half the canvas; the remaining four named vendors share the rest.
Choropleth
WAF adoption is near-universal globally — Japan at 86% is the outlier, almost every country sits above 92%
WAF detection runs at 93% globally, with remarkable consistency across all geographies. Japan is the clear outlier at 86% — Taiwan and Macao also dip below 91%. Every other country in the sample sits above 91%, underlining how thoroughly the global CDN and WAF stack has been adopted.
All six regions sit within one percentage point of each other — regional WAF adoption is essentially uniform
All six regions cluster between 92.5% (Asia) and 93.4% (North America, Europe) — a spread of less than one percentage point. The expected infrastructure gap between developed and developing regions simply does not appear in WAF adoption among popular sites.
European WAF adoption is uniformly high — from France at 92% to the Faroe Islands at 95%
Within Europe, WAF adoption is strikingly uniform — almost all countries sit between 92% and 95%, with France and Bulgaria at the lower end and the Faroe Islands at the top. The anticipated east–west divide simply does not exist for this metric.
Gauge
Japan's WAF adoption sits 7 pp below the global average
Japan is the clearest outlier in the global WAF landscape — its 86% adoption rate is the only major economy that falls meaningfully below the near-universal 93% global benchmark.
Real estate sites average tier 2.3 — well above the global norm
At tier 2.3, the average real estate page requires more than basic HTTP access — JavaScript rendering and light antibot handling are the norm, not the exception.
Heatmap
WAFs dominate every sector; CAPTCHA load is heaviest in Finance and Travel
Across all six sectors, WAF deployment is nearly universal. CAPTCHA and bot scoring diverge sharply by industry — Finance and Travel impose challenges at twice the rate of News and Gov't sites.
Multi-row stacked horizontal
Real estate and entertainment require the most sophisticated access infrastructure
Most industries are overwhelmingly tier 1, but Real Estate and Entertainment stand out — nearly a quarter of their pages require tier 3 or above, reflecting heavy use of bot-detection and dynamic rendering.
robots.txt
GPTBot and CCBot face the most blocks — but search-linked AI agents attract more allows
The disallow/allow ratio reveals commercial intent: ChatGPT-User and OAI-SearchBot attract blocks and explicit whitelists in roughly equal measure, reflecting their value as AI-powered search referrers. CCBot and Bytespider are blocked almost without exception — sites that name them have only one thing to say.
Media and entertainment face the sharpest AI access decisions
Media & Publishing leads not because it is the most broadly defensive, but because its commercial asset — published content — is directly threatened by AI training pipelines. Financial services sits near the bottom: its value lies in transactional services, not extractable text.
Most sites with robots.txt have made a deliberate access decision
56.9% of robots.txt sites are Advanced — they've made a one-sided access decision (either a whitelist or a blacklist, but not both). Only 11.2% have published nothing beyond wildcard rules. The 8.5% with Expert status actively manage access from both ends simultaneously.
Travel and media publish the most deliberate robots.txt policies
Travel & Hospitality has the highest Expert rate of any industry — 15.4% of its robots.txt sites actively manage both whitelists and blacklists simultaneously. Media & Publishing follows closely, consistent with its dual need to stay accessible to search engines while excluding AI training crawlers.