The web is being priced, not blocked

Theresia Tanzil ran the data collection and analysis behind State of Web Access 2026, the largest audit of web access controls ever published: 24,898 sites, 230 countries, 110 industries.

But ask her to sum up that scale succinctly and she doesn't just reach for a number; she highlights the big economic lesson: "The web is being priced up."

"The spirit of the worldwide web was that information needs to be free. But now the information is out there; it's just not free. There's a different cost to getting it."

"That's the key dynamic," she adds. "That's the state of the web in 2026."

Where the effort actually goes

Although only 18.5% of the audited landing pages had no detectable access barrier, 30.7% had just one, and 88% still fell into the Simple or Easy Zyte API access tiers, meaning it can be "quite easy" to scrape most sites using the right tools, Theresia says.

"It's almost like the Pareto ratio, 80/20," she explains. "Eighty percent could be easy. But most of the work is going to be done on the 20%, which is getting more and more aggressive, more and more closed.

"If you're getting paid to do web scraping, most likely you'll be working in that 20%. That's where the work is."

The average job looks different, because nobody pays a data team to go fetch the easy 80%. That's also why the difficulty skews so hard by sector:

• Fashion, Zyte's single hardest industry in the report, needs Moderate-or-harder infrastructure on 57% of its sites.
• Reference and publishing sites sit at the opposite end.

The barrier a developer actually meets has less to do with "the web" in the abstract than with which corner of it pays the bills.

What 'priced up' means for your stack

If the difficulty is concentrated rather than spread evenly across the web, the real question becomes what it costs to sit inside that 20%. According to Tanzil:

"Web scraping at scale is getting more and more complex. It's not just a script kind of problem; it needs to be a system."

"(You need) a system where you need proxy management, you need concurrency, you need to think about whether the page needs JavaScript and what your fallback plan is, how you manage the cost across all of that. It's all about orchestration and instrumentation."

None of that is new to anyone who's shipped a crawler past the prototype stage. What the report adds is a way to price the difficulty in advance, rather than discover it the hard way three weeks into a sprint.

Behind the barriers

Zyte mapped every landing page it audited against its own five-tier scraping-difficulty scale (Simple through Advanced), so "how hard is this site" stops being a question you answer empirically and becomes one you can ask up front.

Six of the seven checks map onto specific pieces of that stack: WAF, antibot, CAPTCHA, JavaScript, rate limiting, and TLS fingerprinting.

The seventh, a non-technical check on robots.txt, stands in for what Tanzil calls "a posture from the website: we prefer you not crawl us this way."

Not every signal is even intended as a "barrier".

• Plenty of sites run JavaScript purely for the sake of user experience. "But at the same time," she says, "it is hindering automated access in some way. It's making it more costly. You need a headless browser."
• The signals don't all behave the same way, mind. CAPTCHA is close to binary: "We want you to be gone." JavaScript, similarly, either loads or it doesn't.
• Rate limiting is a different animal: a dial, not a switch. Zyte's team tested it by sending evenly spaced requests and watching the response codes, then escalating to bursts to see where a site actually pushed back. "It's a gradient," she says. "It's saying: please, yes, you can access this automatically, but don't rush."

How to read the barriers

If there's one operational takeaway here for a developer scoping a project, it's this: don't ask "is this site blocked?," ask "how many of these six mechanisms is it running, and which kind?"

A CAPTCHA changes your answer. A bit of client-side JavaScript might not.

The number that surprised even Zyte

Ask a researcher what surprised them and you usually get a hedge; but Theresia doesn't.

"One surprising finding during the analysis was that anti-bot adoption is only 18.5%," she says. "That wasn't my instinct. At Zyte, we see anti-bot here and there on most of the sites we deal with."

But there's a good reason for that: Zyte's day-to-day book of business skews toward exactly the sites that do run anti-bot software.

"Once you look at what's inside that 18.5%, it makes sense," Theresia adds. "It's sites with e-commerce, pricing intelligence, data that correlates deeply with commercial value."

There's a specific risk in vendor research: the sample a company sees day to day is never a random sample of the web, and Tanzil built this audit specifically to escape that bias rather than confirm it. She's candid about it too, that she likes the moments the data pushed back.

"There were findings I had no expectation of. I didn't even have the question," she says. "But by the time I was looking at the data, I could see the distribution of different anti-bot solutions across different industries.

"It's really interesting when the data speaks to you, instead of you looking for something specific in it."

How the data actually got made

Tanzil built the site list from a list of global and per-country website popularity rankings, which meant wrestling with methodology questions during the research design.

The territory is not a map

"A website that's popular in the US can also be popular in Brazil," she says, "and if it's difficult to access, that counts as a difficult site from a Brazilian perspective too." Even though the site isn't Brazilian at all, she calls that a nuance easy to miss on a first read, but not one that undermines the finding: the difficulty is still what a Brazil-based developer would actually hit.

On the same page

The bigger constraint, though, was scope. Every "website" in the dataset is really a landing page, hit once, from one IP address: a deliberate line drawn against a computational wall.

"If you think about running it from 110 different regional IPs for every site, that's just computationally prohibitive," she says. It's an honest trade-off, openly stated: barriers living deeper in a site, or geo-specific barriers invisible from a single vantage point, aren't in this dataset. What's measured is the front door, not the whole building.

Custom research tooling

The most developer-relevant story, though, is the pivot from tool to system. Zyte's content team started from caniscrape, an open-source CLI that runs a handful of access checks against one URL at a time: a fine exploratory tool, unsuited to an audit running into six figures of URLs.

"I needed to customize that tool and turn it into a Scrapy spider," she says. "I needed to think about how to architect it: partition the whole thing, turn it upside down, break it apart, and pull it back together, so it could take a list of a thousand URLs, one spider, doing just one check."

What came out the other end was, in her description, "a whole bunch of data engineering pre-processing, and then a whole bunch of post-processing" to reassemble an exploded set of per-check rows into something one team could actually sit down and read.

"That's an interesting data engineering challenge, I think, for developers," she adds.

Declining the magic key

The research also swerved relying on Zyte's own Zyte API - for good reason.

"We deliberately didn't use Zyte API," Tanzil says, "because Zyte API is built by design to manage the access and ensure the access is gotten."

"If you run a URL through it, whatever barrier is in front of the page just gets solved."

That's exactly the outcome the audit couldn't afford. "We wanted to capture the web as it is, just a simple request and response, and not taint the process of getting it," Theresia adds. Scrapy Cloud handled orchestration for the project, however.

Where this leaves you

None of this reduces to one tidy number, and that's rather Theresia's point.

Web access in 2026 isn't one big wall. It's a market: prices set unevenly by sector, paid in infrastructure rather than cash. Most of what you'll hit is actually cheap.

The difficult work lives in the expensive fifth, and that fifth just keeps getting pricier.