Data collection
Which sites we looked at
We used SEMrush traffic rankings as our source — the most-visited websites as measured by real web traffic, not a hand-curated list.
Two separate datasets were built:
Global dataset — 9,774 unique websites drawn from SEMrush's worldwide trending list, sampled across 111 industry categories at 100 sites each. Because some sites appear in more than one industry category, the dataset contains 11,100 records in total — the figure cited in the global charts, since each industry analysis counts each record once. This gives a single, stable snapshot of the most popular sites on the commercial web and is the source for the global-level figures throughout this report.
Country-level dataset — 22,138 unique websites selected to represent popular sites across 230 countries and territories. The selection was stratified: for each country, SEMrush's local traffic rankings were sliced by industry category, and the top sites from each country × category combination were pooled to a target of around 1,111 sites per country. This means every country is represented across many types of website — retail, finance, news, travel, and roughly 110 others — not just whichever single category happens to dominate local traffic.
Because many high-traffic sites rank in multiple countries, each site was audited once and its result attributed to every country where it ranked. The 22,138 unique websites produce 216,327 country-level records in total — each representing one site's result in one country. This dataset powers all regional and country-by-country analysis in the report.
Across both datasets combined, 24,898 unique websites were audited — around 7,000 appearing in both. All sites were visited once in October 2025, from a clean session with no prior browsing history or persistent cookies.
How country samples were built
For each country we pulled SEMrush's "most-visited sites" list broken down by category — travel, retail, finance, news, and roughly 110 others. The top sites from each country–category combination were selected and pooled, up to a limit of around 1,111 URLs per country.
This approach — called stratified sampling — ensures every country is represented across many types of website, not just whichever single category happens to be most popular there.
Two important points:
"Country" means popularity, not origin. A site appears under a country because it's popular there, not because the company behind it is based there. Amazon appears in most countries' samples; it doesn't mean Amazon is local to each.
Small territories have smaller samples. Places like Vatican City, Svalbard, or San Marino simply don't have enough popular websites across enough categories to reach 1,111. Their findings are still valid but based on a smaller slice of the web.
Where scans ran from
All scans were run from Zyte's cloud infrastructure. Requests came from data-centre IP addresses — not from residential or mobile connections.
This has one meaningful implication: some sites treat data-centre traffic differently from browser users. A site that applies IP-reputation checks may be more likely to show a CAPTCHA or activate rate limiting for a data-centre request than it would for a typical visitor. Barriers detected this way are still real barriers — they just may not trigger for every visitor in the same way.
The scan origin did not rotate geographically. Every site in the dataset was visited from the same infrastructure environment, so country-to-country differences in the results reflect the sites themselves, not variation in where the scans came from.
What we measured
The access barriers we measured
| Barrier | Full name | What it means |
|---|---|---|
| WAF | Web Application Firewall | A filter that sits between visitors and the web server, screening requests for suspicious patterns. Nearly universal — most sites get one automatically through their CDN or hosting provider. |
| Antibot | Antibot / bot manager | A dedicated system — from vendors like Cloudflare, DataDome, or HUMAN — that analyses visitor behaviour to distinguish real users from bots, often using fingerprinting and machine learning. |
| CAPTCHA | CAPTCHA challenge | A human-verification puzzle — reCAPTCHA, hCaptcha, Turnstile — presented either on every visit or when behaviour looks automated. |
| JavaScript | JavaScript requirement | Content or functionality that only appears after the browser has executed JavaScript. Sites built on frameworks like React or Vue often fall into this category. |
| Rate Limiting | Rate limiting / throttling | The server detects too many requests arriving too quickly and starts refusing them — typically responding with an HTTP 429 ("Too Many Requests") error. |
| TLS | TLS fingerprint filtering | The server checks the technical signature of the connection itself. Automated tools use a different signature from real browsers — sites can block on this difference alone, before any page content is requested. |
| Policy signals | ||
| robots.txt | Standard crawl rules | A standard text file that tells crawlers which parts of a site they're allowed to access. We recorded whether one was published, what rules it contained, and whether it included any crawl-delay directives. Unlike the six barriers above, this doesn't block access — it states preferences. |
| AI blocking | AI crawler blocking | Explicit rules in robots.txt targeting known AI training and scraping agents — GPTBot, ClaudeBot, Bytespider, and others. We recorded the proportion of sites that actively block at least one AI crawler user-agent, and which agents were most commonly targeted. |
Each of the six barriers represents a different technical mechanism a website can use to control or restrict automated access. They were detected independently for every site in the dataset. The two policy signals — robots.txt and AI crawler rules — reflect stated intent rather than active enforcement; a site can publish restrictive robots.txt rules while having no active barriers at all, or vice versa.
How we detected each barrier
How each barrier was detected
| Barrier | How it was detected | Signal strength |
|---|---|---|
| WAF | Sends deliberate probe requests designed to trigger firewall responses, then identifies the vendor from the response signature (using the open-source tool wafw00f). | Good signal |
| Antibot | Looks for known vendor-specific scripts, headers, and challenge pages in the response — each major antibot provider leaves a recognisable footprint. | Good signal |
| CAPTCHA | Checks whether a CAPTCHA widget appears in the page HTML on load, or after a rapid sequence of reloads. The presence of a known widget element (reCAPTCHA, hCaptcha, Turnstile) is a clear signal. | Strong signal |
| JavaScript | Fetches the page twice — once with full JavaScript execution (headless browser) and once without — then compares the content. A significant difference means the site depends on JavaScript to deliver its content. | Good signal |
| Rate Limiting | Sends a controlled burst of requests and watches for HTTP 429 or 403 responses. Because the threshold for triggering rate limiting varies widely by site, some sites that do rate-limit may not have triggered it during the scan. | Partial signal |
| TLS | Sends two requests in sequence — one using a Python HTTP library's default TLS signature, one using a signature that closely mimics a real browser. A different response (or a block) on the first but not the second indicates TLS-based filtering. | Strong signal |
| Policy signals | ||
| robots.txt | Fetches /robots.txt from each domain and parses the file deterministically — recording which User-agent groups exist, which paths are Disallowed, and whether any Crawl-delay directives are present. | Strong signal |
| AI blocking | Checks the parsed robots.txt for Disallow rules that target known AI crawler User-agent strings — including GPTBot, ClaudeBot, CCBot, Bytespider, and others. Exact string matching; no inference. | Strong signal |
WAF was the only barrier that required sending deliberate probe requests. Every other barrier was detected by observing what naturally happened during a standard visit. A barrier is recorded as present only when a clear signal appeared — not seeing a signal doesn't prove the barrier is absent; it may simply not have been triggered during the scan.
Tools used in the scan
| Tool | Version | What it was used for |
|---|---|---|
| Zyte API | — | Source of the scraping difficulty tiers used throughout this report. Each site was tested at both HTTP and browser tiers; the higher of the two was taken as the site's recommended tier. Zyte API's tier model — calibrated against real extraction costs across millions of pages — is the report's primary difficulty metric. |
| Scrapy Cloud | — | Zyte's cloud spider platform. The barrier-detection spider (caniscrape) was deployed and run here, crawling the full dataset and writing results to CSV. |
| caniscrape | Custom fork | Zyte's customised fork of the open-source caniscrape spider. Orchestrated all barrier-detection passes — dispatching each analyser, managing clean sessions, and flattening results into one row per domain. |
| Playwright (Chromium) | Oct 2025 | Headless browser used for JavaScript rendering, CAPTCHA detection (monitoring for reCAPTCHA, hCaptcha, and Turnstile widgets), fingerprinting detection, and the JS-vs-no-JS content comparison. |
| aiohttp + curl_cffi | Oct 2025 | Two Python HTTP clients used together for network probes. aiohttp sends requests with a standard Python TLS signature; curl_cffi sends the same request impersonating a real browser's TLS profile. The difference in server response between the two is how TLS fingerprint filtering is detected. Both are also used in rate-limit probing. |
| wafw00f | 2.3.2 | Open-source WAF fingerprinting tool. Sends crafted probe requests and identifies the firewall vendor from the response signature. The only tool in the stack that deliberately triggers defences rather than passively observing a normal visit. |
Scraping difficulty tiers
How scraping difficulty tiers are assigned
| Tier | What it means | What you need |
|---|---|---|
| 1Simple | A plain HTTP request works — no browser, no special setup. | Standard HTTP client. No browser needed. |
| 2Easy | The site requires JavaScript to render its content, so a headless browser is needed. | Headless browser (e.g. Playwright or Puppeteer). |
| 3Moderate | The site checks connection signatures — a headless browser is needed, configured to look like a real one. | Browser with a realistic fingerprint — matching TLS signature, headers, and canvas behaviour. |
| 4Complex | The site analyses detailed behavioural signals — timing, navigation sequences, event patterns — and blocks sessions that look automated. | Realistic session behaviour, session management, and IP rotation. |
| 5Advanced | The site uses specialist anti-bot infrastructure that requires dedicated countermeasures to access reliably at scale. | Dedicated anti-bot infrastructure; high per-page cost. |
Tiers reflect Zyte API’s real-world pricing model, which is calibrated against the actual compute cost of successfully accessing a site at scale. A higher tier means more infrastructure — and more expense — is required. The "recommended tier" used throughout this report is the more demanding of the two configurations tested for each site: HTTP-only or headless browser. It represents what a scraper would actually need in production.
Geographic regions
How countries are grouped into regions
| Region | Coverage and edge cases |
|---|---|
| Europe | All conventionally European countries, plus Russia and Cyprus. Russia is grouped with Europe rather than Asia because its internet infrastructure, regulatory environment, and web ecosystem are most comparable to European peers. Cyprus is included in Europe by geographic convention. |
| Asia | All countries in continental Asia, plus Turkey. Turkey straddles two continents but is grouped with Asia for this report based on the majority of its territory and the regional comparisons most useful for web access analysis. |
| North America | The United States, Canada, Mexico, Central America, and the Caribbean. Greenland is grouped with North America by geographic convention, despite its administrative ties to Europe. |
| South America | All 12 sovereign countries on the South American continent, plus associated territories. |
| Africa | All 54 African Union member states plus other territories on the African continent. North African countries (Egypt, Morocco, Tunisia, etc.) are grouped with Africa, not with the Middle East. |
| Oceania | Australia, New Zealand, Papua New Guinea, and the island nations of the Pacific — Melanesia, Micronesia, and Polynesia. |
Regional groupings are used to summarise patterns across countries that share geographic proximity and, often, regulatory or infrastructure characteristics. They are a convenience for presentation — country-level data is available throughout the report for more precise comparisons.
Scope and limitations
We only looked at landing pages
Every site in the dataset was visited at its root URL only — the homepage. Sub-pages, product pages, search results, and content behind a login were not scanned.
This matters because access controls are not uniform across a site. A homepage might be freely accessible while a data-rich sub-section is heavily protected — or the reverse. The figures in this report describe the front door, not the whole building.
For most purposes this is the right starting point: homepages reflect the public face of a site's access policy. But anyone planning to scrape specific content types should expect the picture to differ on deeper pages.
Each site was visited once
The scan made a single visit to each site. Some access barriers only activate after repeated requests — rate limiting in particular is designed to respond to volume, not individual visits.
A site that would block a sustained scraping operation may show no barrier at all on a first visit. The rate limiting figures in this report are therefore likely an undercount — the true proportion of sites with rate limiting in place is probably higher than the scan detected.
The other five barriers — WAF, Antibot, CAPTCHA, JavaScript, and TLS — can all be detected on a first visit. Their figures are not affected by the single-visit limitation in the same way.