One in five sites enforces a request threshold — most trigger a block on the very first automated request
| of sites trigger rate limiting | |
|---|---|
| Rate Limiting Adoption | 22.1% |
Rate limiting nominally caps request frequency, but the survey found that 94% of rate-limited sites triggered a block on the very first request even at a one-second interval — effectively zero-tolerance access control rather than graduated throttling. The remaining 6% required a rapid concurrent burst to trigger. Unlike WAF or TLS fingerprinting, the response is always visible in an HTTP error code, making detection unambiguous for crawlers on both sides.
Blocking profile
Nine in ten rate-limited sites return 403 Forbidden — not the purpose-built 429 Too Many Requests
| Share of blocks | |
|---|---|
| 403 Forbidden | 92.6% |
| 429 Too Many Requests | 5.7% |
| 503 Service Unavailable | 1.2% |
| 401 Unauthorized | 0.4% |
HTTP 429 Too Many Requests was designed precisely for this scenario, yet only 6% of sites use it. Most deployments return 403 Forbidden — a generic access-denied code that reveals nothing about a quota window or threshold. Without a 429, no Retry-After header is sent, so a crawler receives no signal about when — or whether — it can try again.
Rate limiting almost always blocks on the first request — severity is near-universal
| Value | |
|---|---|
| Severe — blocked on 1st request | 93.7% |
| Burst-based — concurrent burst only | 5.2% |
| Moderate — 2–4 sequential requests | 1.1% |
The severity distribution is strikingly consistent across industries — when a site deploys rate limiting, it almost always deploys it at maximum aggression. Severe sites block the very first request at a one-second interval, allowing no crawl budget whatsoever. The 5.2% Burst-based tier triggers only under rapid concurrent load, suggesting a different threat model: protecting against DDoS-style hammering rather than sequential crawling. Scrapers should treat any 403 response as a potential rate-limit block and implement backoff strategies accordingly.
Security posture
Rate-limited sites run TLS fingerprinting twelve times more often — a distinctly high-security stack
| Rate-limited sites | All other sites | |
|---|---|---|
| WAF | 99% | 90.5% |
| JavaScript | 53.6% | 36.9% |
| CAPTCHA | 40.8% | 17.4% |
| Antibot | 51.3% | 9.1% |
| TLS | 48.6% | 3.9% |
TLS fingerprinting (49% vs 4%) and Antibot (51% vs 9%) show the sharpest contrasts — rate-limited sites are five to twelve times more likely to layer these detection methods on top. The implication: rate limiting rarely operates alone; it appears as one component in a deliberately constructed, multi-layered access-control stack.
By industry
Fashion leads at 54% — consumer-facing commerce with high-value inventory drives rate limiting adoption
| Value | |
|---|---|
| Apparel & Fashion | 54% |
| Sporting Goods | 49% |
| Jewelry & Luxury | 42% |
| Photography | 42% |
| Restaurants | 41% |
| Furniture | 40% |
| Real Estate | 39% |
| Automotive | 38% |
| Customer Services | 38% |
| Gambling | 35% |
| Veterinary | 35% |
| Adult Content | 34% |
| Wellness | 34% |
| Computer Hardware | 33% |
| Retail | 33% |
The top cohort shares a clear pattern: products with volatile, time-sensitive pricing — apparel, jewellery, sporting goods, furniture, restaurants — where competitor price-scraping or inventory monitoring creates a direct commercial threat. Rate limiting here is a competitive defence, not a security one.
Chemicals and professional services sit at the floor — sectors with little to gain from access control
| Value | |
|---|---|
| Security Products | 15% |
| Foreign Language | 14% |
| Public Relations | 14% |
| Plastics & Polymers | 13% |
| Printing | 13% |
| Banking | 12% |
| Facilities Services | 12% |
| Graphic Design | 12% |
| Public Policy | 11% |
| Writing & Editing | 11% |
| Comics & Animation | 10% |
| Nanotechnology | 10% |
| Accounting | 8% |
| Consulting | 8% |
| Chemicals | 7% |
Chemicals, consulting and accounting sit well below the global average — sectors where the legitimate audience (regulators, investors, partners) expects unrestricted access and where a hard 403 response would cause reputational damage. Banking's 12% reflects a related logic: the most sensitive operations happen behind authenticated sessions, not on landing pages.
By geography
Rate limiting adoption is broadly consistent worldwide, with e-commerce markets pulling slightly higher
| Value | |
|---|---|
| Afghanistan | 12.5% |
| Åland Islands | 11.7% |
| Albania | 14.2% |
| Algeria | 12.7% |
| American Samoa | 11.4% |
| Andorra | 11.7% |
| Angola | 12.3% |
| Anguilla | 11.3% |
| Antigua and Barbuda | 14.7% |
| Argentina | 12.7% |
| Armenia | 13.6% |
| Aruba | 14.3% |
| Australia | 13.5% |
| Austria | 11.6% |
| Azerbaijan | 12.7% |
| Bahamas | 16.2% |
| Bahrain | 12.8% |
| Bangladesh | 13.8% |
| Barbados | 14.3% |
| Belarus | 14.1% |
| Belgium | 12.2% |
| Belize | 14.7% |
| Benin | 12% |
| Bermuda | 14.6% |
| Bhutan | 11.1% |
| Bolivia | 12.6% |
| Bosnia-Herzegovina | 13.3% |
| Botswana | 11.9% |
| Brazil | 12.9% |
| British Indian Ocean Terr. | 13.5% |
| British Virgin Islands | 15.3% |
| Brunei | 12.2% |
| Bulgaria | 13% |
| Burkina Faso | 11.6% |
| Burundi | 10.9% |
| Cabo Verde | 10.4% |
| Cambodia | 13.1% |
| Cameroon | 12.6% |
| Canada | 13.9% |
| Cayman Islands | 14.4% |
| Central African Republic | 9.1% |
| Chad | 13.3% |
| Chile | 12.4% |
| China | 13.6% |
| Colombia | 13.5% |
| Comoros | 10.5% |
| Congo | 12.3% |
| Cook Islands | 14% |
| Costa Rica | 14.1% |
| Côte d'Ivoire | 13% |
| Croatia | 13.1% |
| Curaçao | 14.7% |
| Cyprus | 13.1% |
| Czechia | 12.5% |
| Denmark | 12.1% |
| Djibouti | 11.5% |
| Dominica | 13% |
| Dominican Republic | 14.3% |
| DR Congo | 11.8% |
| Ecuador | 12.3% |
| Egypt | 13.1% |
| El Salvador | 12.5% |
| Equatorial Guinea | 11.6% |
| Eritrea | 9.8% |
| Estonia | 12.7% |
| Eswatini | 13.3% |
| Ethiopia | 13.1% |
| Falkland Islands | 9% |
| Faroe Islands | 10.1% |
| Fiji | 12.5% |
| Finland | 12% |
| France | 12.1% |
| French Polynesia | 11% |
| Gabon | 11.4% |
| Gambia | 11.9% |
| Georgia | 13.4% |
| Germany | 12.2% |
| Ghana | 13.9% |
| Greece | 12.8% |
| Greenland | 12.4% |
| Grenada | 13.7% |
| Guam | 13.5% |
| Guatemala | 13.9% |
| Guernsey | 14.5% |
| Guinea | 12% |
| Guinea-Bissau | 11.9% |
| Guyana | 13.8% |
| Haiti | 13.8% |
| Honduras | 13.7% |
| Hong Kong SAR | 13.8% |
| Hungary | 11.8% |
| Iceland | 13.1% |
| India | 13.2% |
| Indonesia | 13.6% |
| Iraq | 14.5% |
| Ireland | 13% |
| Isle of Man | 13.7% |
| Israel | 13.3% |
| Italy | 12.2% |
| Jamaica | 14.7% |
| Japan | 10.3% |
| Jersey | 13.1% |
| Jordan | 13.5% |
| Kazakhstan | 13.6% |
| Kenya | 14.3% |
| Kiribati | 7.3% |
| Kuwait | 13.5% |
| Kyrgyzstan | 12.4% |
| Laos | 11.9% |
| Latvia | 13.3% |
| Lebanon | 13.9% |
| Lesotho | 12.1% |
| Liberia | 11.9% |
| Liechtenstein | 11.1% |
| Lithuania | 13.1% |
| Luxembourg | 12.4% |
| Macao SAR | 12.3% |
| Madagascar | 13.1% |
| Malawi | 12.2% |
| Malaysia | 14.4% |
| Maldives | 11.6% |
| Mali | 10.4% |
| Malta | 12.6% |
| Marshall Islands | 11.9% |
| Mauritania | 12.8% |
| Mauritius | 12.9% |
| Mexico | 13.9% |
| Micronesia | 13% |
| Moldova | 13.6% |
| Monaco | 14.6% |
| Mongolia | 14.4% |
| Montenegro | 11.6% |
| Montserrat | 11.3% |
| Morocco | 12.8% |
| Mozambique | 11.6% |
| Nauru | 10.4% |
| Nepal | 13.5% |
| Netherlands | 12.6% |
| New Caledonia | 10.3% |
| New Zealand | 13.3% |
| Nicaragua | 14.4% |
| Niger | 11.6% |
| Nigeria | 13.3% |
| Norfolk Island | 10.3% |
| North Macedonia | 13% |
| Northern Mariana Islands | 15.5% |
| Norway | 13.2% |
| Oman | 13.4% |
| Pakistan | 13.8% |
| Palau | 15.7% |
| Palestine | 13.6% |
| Panama | 14.1% |
| Papua New Guinea | 12.3% |
| Paraguay | 12.9% |
| Peru | 13.6% |
| Philippines | 14.3% |
| Poland | 11% |
| Portugal | 13.2% |
| Puerto Rico | 16% |
| Qatar | 12.5% |
| Romania | 13.6% |
| Russia | 12.2% |
| Rwanda | 12.4% |
| Saint Barthélemy | 10.5% |
| Saint Helena | 8% |
| Saint Kitts and Nevis | 14% |
| Saint Lucia | 13.9% |
| Saint Martin | 14.6% |
| Saint Pierre-Miquelon | 10.2% |
| Saint Vincent | 13.2% |
| Samoa | 13.2% |
| San Marino | 13.1% |
| São Tomé and Príncipe | 13.8% |
| Saudi Arabia | 11.9% |
| Senegal | 11.9% |
| Serbia | 13.1% |
| Seychelles | 12.2% |
| Sierra Leone | 11.7% |
| Singapore | 12.8% |
| Sint Maarten | 14.8% |
| Slovakia | 12.7% |
| Slovenia | 12.4% |
| Solomon Islands | 11.8% |
| Somalia | 11.7% |
| South Africa | 12.6% |
| South Korea | 13.6% |
| South Sudan | 9.1% |
| Spain | 11.9% |
| Sri Lanka | 12.6% |
| Suriname | 13.8% |
| Sweden | 12.1% |
| Switzerland | 12.4% |
| Taiwan | 13.2% |
| Tajikistan | 12.5% |
| Tanzania | 13% |
| Thailand | 13.1% |
| Timor-Leste | 12% |
| Togo | 10.4% |
| Tonga | 12% |
| Trinidad and Tobago | 13.8% |
| Tunisia | 13% |
| Turkey | 13% |
| Turkmenistan | 12.5% |
| Turks and Caicos | 15.8% |
| Uganda | 11.9% |
| Ukraine | 13.8% |
| United Arab Emirates | 12.7% |
| United Kingdom | 13.1% |
| United States | 14.9% |
| Uruguay | 12.9% |
| US Virgin Islands | 15.1% |
| Uzbekistan | 12.5% |
| Vanuatu | 11.3% |
| Vatican City | 11.5% |
| Venezuela | 14.3% |
| Vietnam | 12.8% |
| Wallis and Futuna | 10.1% |
| Yemen | 12.6% |
| Zambia | 12% |
| Zimbabwe | 11.8% |
Country-level rates cluster between 7% and 17% — meaningful variation at the country level, but far narrower than the industry spread (7% to 54%). No region opts decisively out; the modest geographic gradient likely reflects differences in the share of e-commerce and consumer-retail sites in each country's sampled web footprint.
By site scale
Adoption rises with scale — large sites are most likely to rate-limit, with a slight dip at the top tier
| Rate limiting adoption | |
|---|---|
| Micro (<350K/mo) | 14.5% |
| Small (350K–1.1M) | 20.9% |
| Mid (1.1M–4.2M) | 23% |
| Large (4.2M–21.5M) | 26.9% |
| Major (21.5M+) | 25.2% |
Rate limiting is a capacity and revenue-protection decision as much as a security one. Low-traffic sites rarely face the scraping pressure that justifies the operational overhead of deploying and tuning rate limits. As sites grow into the tens of millions of monthly visits, the commercial value of their data — and the cost of uncontrolled crawler traffic — rises with scale — though the Major tier (21.5M+ visits) dips slightly below the Large tier, suggesting the very largest sites rely on other layers of their security stack.