AI changes three things about web scraping and leaves a fourth alone.
Collecting data from the web has meant writing and running a scraper. But old scraping tools are starting to show their age as they try to keep up with a web landscape that is growing more complex and defensive. Think about an engineering team that has to spend a lot of time updating scrapers every time a website's layout changes, or has to fight complex anti-bot systems.
AI changes extraction, because a model can read a page and return fields without a selector written for that site. It changes adaptation, because a layout change that breaks a selector need not break a model. And it changes the cost of breadth, because adding a source no longer means writing and maintaining a parser for it.
It does not change access. Getting the page in the first place is a different problem, solved by different means, and a model that cannot see a page cannot extract anything from it.
That distinction runs through the rest of this page.
Traditional or AI extraction: which should you use?
For anyone who has managed a web scraping project, the pain points are familiar. Traditional scrapers are built on hard-coded rules, typically methods such as CSS selectors or XPaths, to find data. This makes them extremely brittle. A renamed CSS class can break the rule that reads it. This forces developers into a frustrating and costly cycle of monitoring, debugging, and patching, making it nearly impossible to maintain reliable data extraction from hundreds of sources.
| Selectors you write | A model that reads the page | |
|---|---|---|
| How fields are identified | A rule per site, written by you | Inferred from the page |
| Cost of adding a source | One parser, written and maintained | Close to zero |
| What a layout change does | Breaks the selector, usually silently | Can survive it |
| Cost per page | Almost nothing once written | Real, and charged every run |
| Predictability | Exact. The same selector returns the same field | Probabilistic. Needs validation |
| How it fails | Returns nothing, or returns the wrong element | Returns a confident wrong value |
| Getting past defences | Unchanged | Unchanged |
| Best for | A few stable sources at high volume | Many sources, or sources that change |
Neither column helps with being blocked, because neither column is what makes the request.
Why does a model not get you past a block?
Because the block happens first.
A defended site decides whether to serve you before any content exists to read. It decides on your address reputation, your request headers, your TLS handshake characteristics, your browser fingerprint and your request rate. All of those are properties of the fetch. What happens to the bytes afterwards is invisible to the site.
Evading these is a constant cat-and-mouse game that can take your team's focus from data extraction to deal with less important ban related issues.
So a model does not help you get in. What genuinely helps is address diversity, a real browser where one is required, request rates that do not trip a limit, and honouring what the site tells you. A site asking you to slow down says so with a 429 status, which "indicates that the user has sent too many requests in a given amount of time", and may carry a Retry-After header (RFC 6585, April 2012).
This matters more each year rather than less. Across the HTTP Archive corpus, 33% of desktop sites and 32% of mobile sites were running a bot protection service in 2024, up from 29% and 26% in 2022 (Web Almanac 2024, Security chapter).
Rendering is part of the same layer, and it is not free. Modern websites rely heavily on JavaScript to load content asynchronously. A basic scraper that only fetches the initial HTML will constantly miss crucial data. While headless browsers like Playwright can render this content, they are slow and resource-intensive. Running and managing these browsers at scale is traditionally a logistical nightmare.
Where AI does appear in the access layer, it is doing something different from extraction: choosing which address and which browser profile to use, and learning which combination works on which target. That is a useful thing and it is not the same capability. How anti-bots work covers what is being read.
What does AI extraction cost per page?
The published figures let you work it out, so here is the arithmetic rather than an assertion.
The input size. The median homepage carried 18 KB of HTML on both desktop and mobile in the 2024 measurement (Web Almanac 2024, Page Weight chapter). A widely used tokeniser states that "on average, in practice, each token corresponds to about 4 bytes" (tiktoken). At that rate the median homepage's HTML is roughly 4,500 input tokens. Models tokenise differently, so read this as an order of magnitude rather than a measurement of your own pages.
The price. One provider's published list prices input tokens at $1, $2 and $4 per million on three of its current models, with a higher tier above those (Anthropic pricing, API tab).
The multiplication. Input cost only, for 1,000 pages at the median size:
| Input price per million tokens | Input tokens for 1,000 median pages | Input cost per 1,000 pages |
|---|---|---|
| $1 | 4,500,000 | $4.50 |
| $2 | 4,500,000 | $9.00 |
| $4 | 4,500,000 | $18.00 |
Four factors move that number. The first two push it up, the last two pull it down.
- Output tokens are extra, and they are priced higher than input on every model in that list.
- The median is not your pages. The 18 KB figure is a homepage median. Measure your own product and listing pages before you budget, and count the scripts, styles and inline data if you send the page whole.
- Stripping the page first cuts it. Sending only the visible text, or only the region containing the fields, reduces the input tokens you pay for.
- Prices change. Redo the arithmetic against the current list rather than quoting this table in a year.
Now compare that to a written selector, which costs engineering time once and then runs for the price of the fetch. The per-page cost is one part of that comparison. Scaling a traditional scraping operation is a significant infrastructure challenge. The overhead in both cost and engineering time balloons quickly, creating a major bottleneck that prevents businesses from expanding their data collection efforts.
The crossover is a real calculation: at 1,000 pages from one stable source, the selector wins easily. At 1,000 pages from 200 sources that each change quarterly, it does not, because you are now maintaining 200 parsers.
What happens when a model gets a field wrong?
It returns something that looks right.
A selector fails loudly. It matches nothing and you get an empty field, or it matches the wrong element and you get an obviously wrong value, like a delivery date where a price should be. Either way the defect is visible in the output.
A model fails quietly. It returns a plausible price that is the crossed-out original rather than the current one, or the shipping cost, or the price of a related item further down the page. Every row is well formed. Nothing raises an error.
No AI is perfect. It's crucial to monitor the accuracy of extracted data, as models can sometimes misclassify fields. Over time, as websites evolve significantly, a model's accuracy can degrade ("model drift").
So AI extraction needs a validation layer that selector-based extraction does not:
- Type and range checks per field. A price is a number in a plausible range. A date parses and is not in the wrong decade.
- A sample checked by a person, per source, on a schedule. Not once at launch. Model behaviour and page layouts both change.
- Cross-field consistency. A product in stock with no price, or a discount larger than the price, is a flag rather than a row.
- Distribution monitoring. If the average price from one source moved sharply overnight and the market did not, the extraction changed rather than the market.
None of that is a reason to avoid AI extraction. It is the cost that replaces parser maintenance, and it is not zero, so budget for it.
Does it fix matching records across sources?
No.
A model reads one page at a time. It does not know that the product it just read from source A is the same product it read from source B under a different name. Joining those records still needs a stable identifier, a matching rule decided before collection rather than after, and the source and collection date stored on every row.
When is a written selector still the right answer?
Three conditions.
The source is stable. It changed twice in two years, and both times you noticed within a day.
The volume is high. Hundreds of thousands of pages per run, where a per-page cost of any size becomes the largest line in the budget.
The field must be exact. A financial or regulatory field where a plausible wrong answer is worse than a missing one. A selector that breaks tells you it broke.
Under those conditions, a written selector is faster, cheaper per page, and fails in a way you can detect. A production pipeline can run both: selectors on the handful of sources that justify the maintenance, model extraction on the long tail that never would.
What kinds of work does this open up?
The pattern is the same in every case. It becomes worth collecting from many sources rather than a few, because the cost of the two hundredth source is no longer a parser.
- Price and range monitoring across a wide retail set, rather than the handful of competitors somebody had time to write parsers for.
- Aggregating listings in property, recruitment or travel, where every source publishes the same fields in a different structure and the structures change. Aggregator platforms thrive on timely, structured data.
- Alternative data for research, pulling from news, filings, forums and company sites, where breadth is the point and no single source justifies its own parser.
- Brand and reputation monitoring across reviews and forums, where the extraction step feeds a classification step and both can run on the same content.
Two of those carry a data protection question that the technology does not answer. Reviews, forum posts and profiles can identify people, and personal data stays personal data through every processing step. That is a decision to make before collection, not after.
What should you check before adopting it?
While AI offers immense power, it's essential to approach it responsibly. Six checks.
| Check | Why | What good looks like |
|---|---|---|
| Accuracy on your pages, not a benchmark | Performance varies by page type and by site | A measured field-level accuracy on a sample of your own sources |
| The cost at your volume and your page sizes | The median page is not your page | A per-page figure from a real run, not a list price |
| What happens on a page the model has not seen | The long tail is the reason to use it | A stated behaviour for low confidence, rather than a guess returned as a value |
| Whether the access layer is included | Extraction without access is half a system | A single answer for fetching, rendering and unblocking |
| How model changes are handled | A provider updating a model changes your output | A stated versioning and deprecation policy |
| What is stored and for how long | Your target pages pass through somebody else's system | A written data handling position |
Where does Zyte fit?
Zyte sells this.
Zyte API covers both halves behind one endpoint: the access layer, which is address selection, rendering and unblocking, and AI extraction, which returns product, article and other page types as typed fields without a selector written for the site. The reason those are sold together is the argument at the top of this page. An extraction capability with no access layer stops at the first defended source.
Zyte also maintains Scrapy, the open source framework, which remains the right answer for the stable, high-volume sources described above and costs nothing.
If you want none of that, everything on this page still applies. The arithmetic, the validation layer and the join are the same whoever provides the model.
FAQ
Q: How is AI web scraping different from traditional scraping?
A: Traditional scraping identifies each field with a rule written for that site. AI extraction reads the page and returns the fields without one, which makes a new source cheap to add and lets extraction survive some layout changes. It is charged per page, and its output needs checking, because it fails with a plausible wrong value rather than an empty one.
Q: How does AI handle anti-bot systems and IP blocks?
A: It does not. Access and extraction are separate problems. A defended site decides whether to serve a request before there is any content to read, based on the address, the headers, the TLS characteristics and the request rate. Those belong to the fetch. AI can help choose which address and browser profile to use, which is a different capability from reading the page.
Q: What kinds of work does AI web scraping suit?
A: Work where breadth is the point: price and range monitoring across a wide retail set, listings aggregation in property, recruitment or travel, alternative data from news and filings, and brand monitoring across reviews and forums.
Q: Does AI make web scraping easier to scale?
A: It makes adding sources cheaper, because a new source no longer needs a parser. It does not make access cheaper. Addresses, rendering and unblocking cost the same whatever reads the page afterwards.
Q: What should I check before adopting AI web scraping?
A: Accuracy on your own pages rather than on a benchmark, the cost at your volume and page sizes, what the system does with a page it has not seen, whether access is included, how model changes are handled, and what happens to the pages you send.
Q: Is AI scraping more expensive than writing a parser?
A: Per page, yes. A selector costs engineering time once and then runs for the price of the fetch; a model is charged every time it reads a page. Over a project it depends on how many sources you have and how often they change, and the crossover is a calculation rather than a rule.
Q: Does AI make scraping unblockable?
A: No. Nothing does. The block happens before the page is read, so what reads it afterwards makes no difference to whether you got it.
Q: When should I still write a parser?
A: When the source is stable, the volume is high and the field must be exact. Under those three conditions a selector is cheaper, faster and fails in a way you can detect.
Q: How do I know the model extracted the right value?
A: You check. Type and range checks on every field, a sample reviewed by a person per source on a schedule, cross-field consistency rules, and monitoring the distribution of each field so a silent change shows up as a shift rather than an error.
Q: Does AI extraction handle JavaScript-heavy pages?
A: Only if something rendered the page first. The model reads what it is given. Rendering is part of the access layer, and it costs more per page than a plain fetch whoever does it.
Q: Can AI match the same product across different websites?
A: Not on its own. It reads one page at a time and has no knowledge of the other source. Joining records still needs a stable identifier and a matching rule you decide before you collect.
