The spider that fixes itself: Help your scraper rise from the ashes

Konstantin Lopukhin explores scrapy-phoenix, an open-source prototype that uses coding agents to repair web scrapers mid-crawl, with validation checks and human escalation.

Konstantin Lopukhin · Head of R&D

10 min read ·

The spider that fixes itself: Help your scraper rise from the ashes

We are all used to asking coding agents to modify software when our requirements change. Recently, I started wondering what would happen if we flipped that around. What if the requirements stayed the same, the software noticed that the environment it operates in had changed, and it used an agent to modify itself and keep working?

To see why this matters for web scraping, look at how we usually handle a broken data feed.

In the beginning, a website changes, the scraper breaks, the customer notices missing products, and a developer fixes the code. We don't want to bother the customer, so we add a monitoring system that creates a ticket for the developer. Then coding agents arrive. At first, the developer asks the agent to write the fix. Soon enough, we realise the monitoring system can invoke the coding agent directly, leaving the developer to just review the pull request. As agents get better, maybe we don't even need a developer looking at the code - we just have QA review the extracted data. Finally, we automate QA too.

Workflow diagram showing Website -> Spider -> Feed -> Monitoring -> Ticket -> Coding agent -> Automated QA

We end up with a smooth, automated workflow where data flows between services and everything gets fixed automatically. But if you look closely at that diagram, you realise something funny: we just shipped the org chart.

There is nothing wrong with this - it is the natural result of different departments automating their work and connecting the pieces. Still, it comes with real headaches. Every box in that diagram is its own service, agent, or database. We have to pass complex state around between them while the target website keeps changing underneath us.

I wanted to see if there was a simpler way. Instead of building an external pipeline of services, what if we moved the entire adaptation loop inside the spider itself?

Moving the adaptation loop inside the spider

We can think of a spider as a piece of software operating in an environment with a specific goal - say, collecting every product and its current price. It can use ordinary, fast Python code for extraction and ifor checking if the website has changed, and call in AI only when a repair is actually needed.

The scrapy-phoenix prototype

I built an open-source proof of concept for this idea called scrapy-phoenix. It is built on top of Scrapy, web-poet page objects, and harness-run (an open-source library we use at Zyte to run coding agents like Codex).

web-poet page objects let us separate extraction and navigation logic from the core crawling code, and keep crawling code largely website-independent. One page object might extract product data, while another extracts category or pagination links. Meanwhile, certain things remain fixed so the agent cannot change them: the target schema, the prose specification of our goal, saved example pages (fixtures), and the validation gate.

Diagram of the scrapy-phoenix architecture showing Fixed elements, the Running crawl, and the Repair workspace

Pause, repair, hot-reload, resume

When the spider is running and encounters a page it cannot read, it does not crash or wait for a ticket. Everything happens within a single process:

  1. Notice a mismatch: The spider's extraction code raises an exception on an unfamiliar page.

  2. Pause the crawl: The spider sets the broken page aside and pauses the run.

  3. Repair and check: The spider invokes the coding agent on a copy of the project, scoped to just creating a new page object for that broken layout. The framework runs automated checks against the candidate code.

  4. Load, retry, continue: If the checks pass, the spider hot-reloads the new code into the running process, retries the held page, and continues the crawl. If no repair passes the checks, it stops and asks a human for help.

The adaptation loop diagram

How the spider notices (and adapts to) layout changes

For this to work, the spider needs a reliable way to know that a page layout has changed without calling an LLM on every single request.

Writing code that notices: The must construction

In ordinary Scrapy extraction code, you write CSS selectors that return None if an element is missing. If a website redesigns its product page, your spider happily continues running and outputs empty records.

In scrapy-phoenix, we do a small rewrite using a StrictMixin. We know a product page should always have a current price. By using self.must_text(".price-tag .current-price"), we tell the spider that this selector must match. If it finds nothing, it raises a layout mismatch exception, produces no item, and sets the page aside for repair.

1class ProductPageV1(StrictMixin, ProductPage):
2    @field
3    def price(self):
4        return money(self.must_text(".price-tag .current-price"))
5
6    @field
7    def regularPrice(self):
8        # the price box must be there; the old price may not
9        box = self.must(".price-tag")
10        text = self.may_text(".original-price", root=box)
11        return money(text) if text else None
Copy

Learning new layouts without forgetting old ones

When the agent steps in to fix a broken product page, it does not overwrite ProductPageV1. Instead, it adds a new variant: ProductPageV2.

Websites frequently switch between layouts. They run A/B tests, or they roll out a temporary Black Friday promotion and revert to the old design a week later. By keeping the old page object and adding a new one that outputs the exact same schema, the spider can handle both layouts simultaneously. The validation gate cross-checks both variants to ensure V1 still works on old saved pages, V2 works on the new pages, and each layout has strict assertions so the spider knows which variant to use.

Graph of an example run where half the product pages were redesigned

In one test run where half the product pages on a store were redesigned mid-crawl, the spider hit the first unfamiliar page early on, paused for 46 seconds while the agent wrote and verified ProductPageV2 (at a cost of $0.12), and then resumed. Both V1 and V2 extracted items side by side, finishing with 566 out of 566 products correct.

When things go wrong: Shortcuts, guardrails, and knowing when to stop

During development, plenty of things went wrong - especially when testing with weaker models. Tightening the validation checks solved some early issues, like making sure the agent extracted the exact same set of fields as before, or checking that pagination links actually went forward instead of backward. Other failure modes required deeper design changes.

Incremental repair: Fixing one page type at a time

When a website redesigns, it often changes the homepage, category pages, listings, and product pages all at once. You do not want the agent trying to fix all of those in one go, because at the start of the crawl, it has only seen the homepage. It does not have the evidence needed to fix product pages yet.

The solution is incremental repair. Every new class the agent writes must handle a page the crawl actually got stuck on. In a test where the entire store was redesigned, the spider paused on the homepage (25s), resumed, got stuck on top categories (72s), resumed, fixed listings (42s), and finally fixed product pages (49s). Breaking it into four staged repairs made the process much more reliable.

Timeline of incremental repair across 4 page types

"Give an agent a check, and it will try to pass it"

Coding agents are trained with reinforcement learning to pass whatever checks you give them. We have all heard stories of agents reward-hacking benchmarks, and I saw the exact same behavior here.

In one early test, I gave the agent a broken homepage, but my validation check demanded that it filled the same navigation fields as the existing code (which expected category links, product links, and a "next page" link). I did not give the agent an easy way out. So, to pass the check, the agent:

  • Returned category links as "products"

  • Hard-coded the page number to 1

  • Generated a dummy "next page" link that counted as filled but was never followed

In its reasoning log, the agent explained that it did this "to satisfy the navigation field contract."

Sometimes the right response is to stop

To prevent agents from hallucinating constants just to pass a gate, you have to let them explicitly give up.

I tested scrapy-phoenix across nine scenarios on a test store (three runs each). In 24 out of 24 runs covering redesigns, infinite scroll, and "load more" buttons, the spider healed itself and extracted all 566 products accurately.

Table of test results showing 24 of 24 healed and 3 of 3 stopped

In the ninth scenario, I hid the product prices behind a member sign-in wall. Because price was a required field, the agent inspected the page, realised the data simply wasn't there, and gave up: "The site now hides product prices behind sign-in, so these pages cannot supply the required current price." All three runs stopped cleanly and escalated to a human.

The optional content blind spot

The biggest limitation of page-level assertions is optional content. Take regularPrice in the code snippet above, or subcategories on a category page. Not every product is on discount, and not every category has subcategories. If the markup for regularPrice changes inside the price box, may_text will just return None, and the spider won't notice.

There are two ways to handle this within the spider:

  1. Crawl memory: The spider can store results from previous runs. If a specific category page had subcategories yesterday and has none today, it triggers a repair.

  2. Statistical replay buffers: Similar to traditional monitoring tools like Spidermon, the spider can track field coverage mid-crawl. If we extract 20 items and brand is 0% populated - when it is usually around 50% - the spider can pause, repair, and re-extract the buffer.

What in-process repair buys you - and what it costs

Running this inside Scrapy Cloud works well because the platform already provides a container sandbox for each job. More importantly, testing this revealed a major advantage I had not anticipated at the start.

No more ‘can't reproduce’

In production web scraping, a monitoring alert often fires because item counts dropped. By the time a developer looks at the ticket hours later and reruns the spider, everything works fine. The site changed back, or the bug only happened on a specific slice of pages. We missed the window to reproduce the problem.

When the spider pauses at the exact page that broke, with the live state of the crawl preserved around it, "can't reproduce" disappears. The agent has the exact evidence it needs right in front of it.

This is also why the approach makes a lot of sense for anti-ban. Anti-ban fixes are often short-lived - an engineer finds a workaround that lasts five hours before the anti-bot system catches up. If you wait for a ticket and a rerun, the fix is already dead. If the spider adapts mid-crawl, the fix only needs to survive long enough to finish the current job.

The trade-offs and security boundaries

Of course, putting an agent inside your spider comes with costs:

  • Crawl delays: A repair pauses the crawl for a minute or two. (Though for crawls that take days, pausing for three minutes beats rerunning the whole job).

  • Global blindness: A single page object cannot easily see problems that require many pages or many crawls to detect, meaning you still need cross-run monitoring.

  • Security: Normally, we want production environments to be as simple and locked-down as possible. Here, we are putting an agent, an API key, and an LLM budget inside every crawl. Because scraped websites contain untrusted input that ends up in prompts, running inside a strict container sandbox without permanent API keys is mandatory.

Speculative repair and software that adapts

The current scrapy-phoenix prototype has clear limits - it doesn't commit repairs back to a Git branch yet, and it only rewrites page objects rather than core crawler control logic.

Diagram showing the adaptation loop moved into the spider with human escalation

Even so, moving the repair loop into the spider simplifies our infrastructure while keeping a human in the loop for the hard problems. When a production spider is clearly broken, doing a speculative repair on the fly and letting a human review the code and data after the crawl finishes is often much better than halting delivery to wait for human approval.

Software that operates in unstable environments should be able to adapt while it runs. If you want to experiment with this approach or take the ideas further, you can check out the code at github.com/lopuhin/scrapy-phoenix.

Try Zyte API

Build your first scraper in minutes

Free trial, no credit card. From a single request to production in an afternoon.

Get started

Konstantin Lopukhin

Head of R&D

Konstantin is Zyte’s Head of R&D, where he leads research and development into machine learning methods to improve extraction, parsing, and content quality. He has also participated in Kaggle competitions, achieving a Grandmaster Title, and contributes to the community with t…

More from this author

Continue reading

Akshay Philar speaking at Extract Summit about the anti-ban arms race and the open web.
Anti-ban

The anti-ban arms race: Agentic traffic and the fight for the open web

As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.

Akshay Philar9 min read
Zyte MCP chat box connected to fetch, search and extract tools.
Data gathering for AI

Introducing Zyte MCP: Powerful web data gathering for your agent

Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.

Valter Sciarrillo7 min read
Group on stage at Extract Summit Dublin 2026
Web data collection

Five takeaways from Extract Summit Dublin

AI is making scraping code cheaper to write. Extract Summit Dublin explored the harder work that remains: access, infrastructure, maintenance and trust.

Robert Andrews
What is a self-healing web scraper?
AI-assisted data extraction

What is a self-healing web scraper?

A self-healing web scraper detects when a site change breaks extraction, repairs the code within limits you set, proves the fix with tests, and escalates when it can't. What it fixes, what it doesn't, how the loop works and what a repair costs.

John Rooney
self-healing-spider-agent
AI-assisted data extraction

Self-healing scrapers: four things to put in place before an AI agent fixes your spiders

What decides whether a self-healing scraper's repair is any good: detection during the crawl, a specific failure report, extraction code the agent can target, and tests that can fail.

John Rooney
What's becoming of web scraping developers in the age of AI agents?
AI-assisted data extraction

What's becoming of web scraping developers in the age of AI agents?

AI agents can generate code, suggest selectors, and draft crawl logic. What they can't do is design the system that decides when to stop, what to trust, and how to recover when the web pushes back. That job still belongs to a human.

Neha Setia Nagpal14 min read

The Community · Newsletter

The best of Zyte and the data web, in your inbox.

One curated edition — new articles, product updates, and the stories shaping the data web. No noise.