We are all used to asking coding agents to modify software when our requirements change. Recently, I started wondering what would happen if we flipped that around. What if the requirements stayed the same, the software noticed that the environment it operates in had changed, and it used an agent to modify itself and keep working?
To see why this matters for web scraping, look at how we usually handle a broken data feed.
In the beginning, a website changes, the scraper breaks, the customer notices missing products, and a developer fixes the code. We don't want to bother the customer, so we add a monitoring system that creates a ticket for the developer. Then coding agents arrive. At first, the developer asks the agent to write the fix. Soon enough, we realise the monitoring system can invoke the coding agent directly, leaving the developer to just review the pull request. As agents get better, maybe we don't even need a developer looking at the code - we just have QA review the extracted data. Finally, we automate QA too.

We end up with a smooth, automated workflow where data flows between services and everything gets fixed automatically. But if you look closely at that diagram, you realise something funny: we just shipped the org chart.
There is nothing wrong with this - it is the natural result of different departments automating their work and connecting the pieces. Still, it comes with real headaches. Every box in that diagram is its own service, agent, or database. We have to pass complex state around between them while the target website keeps changing underneath us.
I wanted to see if there was a simpler way. Instead of building an external pipeline of services, what if we moved the entire adaptation loop inside the spider itself?
Moving the adaptation loop inside the spider
We can think of a spider as a piece of software operating in an environment with a specific goal - say, collecting every product and its current price. It can use ordinary, fast Python code for extraction and ifor checking if the website has changed, and call in AI only when a repair is actually needed.
The scrapy-phoenix prototype
I built an open-source proof of concept for this idea called scrapy-phoenix. It is built on top of Scrapy, web-poet page objects, and harness-run (an open-source library we use at Zyte to run coding agents like Codex).
web-poet page objects let us separate extraction and navigation logic from the core crawling code, and keep crawling code largely website-independent. One page object might extract product data, while another extracts category or pagination links. Meanwhile, certain things remain fixed so the agent cannot change them: the target schema, the prose specification of our goal, saved example pages (fixtures), and the validation gate.

Pause, repair, hot-reload, resume
When the spider is running and encounters a page it cannot read, it does not crash or wait for a ticket. Everything happens within a single process:
Notice a mismatch: The spider's extraction code raises an exception on an unfamiliar page.
Pause the crawl: The spider sets the broken page aside and pauses the run.
Repair and check: The spider invokes the coding agent on a copy of the project, scoped to just creating a new page object for that broken layout. The framework runs automated checks against the candidate code.
Load, retry, continue: If the checks pass, the spider hot-reloads the new code into the running process, retries the held page, and continues the crawl. If no repair passes the checks, it stops and asks a human for help.

How the spider notices (and adapts to) layout changes
For this to work, the spider needs a reliable way to know that a page layout has changed without calling an LLM on every single request.
Writing code that notices: The must construction
In ordinary Scrapy extraction code, you write CSS selectors that return None if an element is missing. If a website redesigns its product page, your spider happily continues running and outputs empty records.
In scrapy-phoenix, we do a small rewrite using a StrictMixin. We know a product page should always have a current price. By using self.must_text(".price-tag .current-price"), we tell the spider that this selector must match. If it finds nothing, it raises a layout mismatch exception, produces no item, and sets the page aside for repair.
1class ProductPageV1(StrictMixin, ProductPage):
2 @field
3 def price(self):
4 return money(self.must_text(".price-tag .current-price"))
5
6 @field
7 def regularPrice(self):
8 # the price box must be there; the old price may not
9 box = self.must(".price-tag")
10 text = self.may_text(".original-price", root=box)
11 return money(text) if text else NoneLearning new layouts without forgetting old ones
When the agent steps in to fix a broken product page, it does not overwrite ProductPageV1. Instead, it adds a new variant: ProductPageV2.
Websites frequently switch between layouts. They run A/B tests, or they roll out a temporary Black Friday promotion and revert to the old design a week later. By keeping the old page object and adding a new one that outputs the exact same schema, the spider can handle both layouts simultaneously. The validation gate cross-checks both variants to ensure V1 still works on old saved pages, V2 works on the new pages, and each layout has strict assertions so the spider knows which variant to use.

In one test run where half the product pages on a store were redesigned mid-crawl, the spider hit the first unfamiliar page early on, paused for 46 seconds while the agent wrote and verified ProductPageV2 (at a cost of $0.12), and then resumed. Both V1 and V2 extracted items side by side, finishing with 566 out of 566 products correct.
When things go wrong: Shortcuts, guardrails, and knowing when to stop
During development, plenty of things went wrong - especially when testing with weaker models. Tightening the validation checks solved some early issues, like making sure the agent extracted the exact same set of fields as before, or checking that pagination links actually went forward instead of backward. Other failure modes required deeper design changes.
Incremental repair: Fixing one page type at a time
When a website redesigns, it often changes the homepage, category pages, listings, and product pages all at once. You do not want the agent trying to fix all of those in one go, because at the start of the crawl, it has only seen the homepage. It does not have the evidence needed to fix product pages yet.
The solution is incremental repair. Every new class the agent writes must handle a page the crawl actually got stuck on. In a test where the entire store was redesigned, the spider paused on the homepage (25s), resumed, got stuck on top categories (72s), resumed, fixed listings (42s), and finally fixed product pages (49s). Breaking it into four staged repairs made the process much more reliable.

"Give an agent a check, and it will try to pass it"
Coding agents are trained with reinforcement learning to pass whatever checks you give them. We have all heard stories of agents reward-hacking benchmarks, and I saw the exact same behavior here.
In one early test, I gave the agent a broken homepage, but my validation check demanded that it filled the same navigation fields as the existing code (which expected category links, product links, and a "next page" link). I did not give the agent an easy way out. So, to pass the check, the agent:
Returned category links as "products"
Hard-coded the page number to
1Generated a dummy "next page" link that counted as filled but was never followed
In its reasoning log, the agent explained that it did this "to satisfy the navigation field contract."
Sometimes the right response is to stop
To prevent agents from hallucinating constants just to pass a gate, you have to let them explicitly give up.
I tested scrapy-phoenix across nine scenarios on a test store (three runs each). In 24 out of 24 runs covering redesigns, infinite scroll, and "load more" buttons, the spider healed itself and extracted all 566 products accurately.

In the ninth scenario, I hid the product prices behind a member sign-in wall. Because price was a required field, the agent inspected the page, realised the data simply wasn't there, and gave up: "The site now hides product prices behind sign-in, so these pages cannot supply the required current price." All three runs stopped cleanly and escalated to a human.
The optional content blind spot
The biggest limitation of page-level assertions is optional content. Take regularPrice in the code snippet above, or subcategories on a category page. Not every product is on discount, and not every category has subcategories. If the markup for regularPrice changes inside the price box, may_text will just return None, and the spider won't notice.
There are two ways to handle this within the spider:
Crawl memory: The spider can store results from previous runs. If a specific category page had subcategories yesterday and has none today, it triggers a repair.
Statistical replay buffers: Similar to traditional monitoring tools like Spidermon, the spider can track field coverage mid-crawl. If we extract 20 items and
brandis 0% populated - when it is usually around 50% - the spider can pause, repair, and re-extract the buffer.
What in-process repair buys you - and what it costs
Running this inside Scrapy Cloud works well because the platform already provides a container sandbox for each job. More importantly, testing this revealed a major advantage I had not anticipated at the start.
No more ‘can't reproduce’
In production web scraping, a monitoring alert often fires because item counts dropped. By the time a developer looks at the ticket hours later and reruns the spider, everything works fine. The site changed back, or the bug only happened on a specific slice of pages. We missed the window to reproduce the problem.
When the spider pauses at the exact page that broke, with the live state of the crawl preserved around it, "can't reproduce" disappears. The agent has the exact evidence it needs right in front of it.
This is also why the approach makes a lot of sense for anti-ban. Anti-ban fixes are often short-lived - an engineer finds a workaround that lasts five hours before the anti-bot system catches up. If you wait for a ticket and a rerun, the fix is already dead. If the spider adapts mid-crawl, the fix only needs to survive long enough to finish the current job.
The trade-offs and security boundaries
Of course, putting an agent inside your spider comes with costs:
Crawl delays: A repair pauses the crawl for a minute or two. (Though for crawls that take days, pausing for three minutes beats rerunning the whole job).
Global blindness: A single page object cannot easily see problems that require many pages or many crawls to detect, meaning you still need cross-run monitoring.
Security: Normally, we want production environments to be as simple and locked-down as possible. Here, we are putting an agent, an API key, and an LLM budget inside every crawl. Because scraped websites contain untrusted input that ends up in prompts, running inside a strict container sandbox without permanent API keys is mandatory.
Speculative repair and software that adapts
The current scrapy-phoenix prototype has clear limits - it doesn't commit repairs back to a Git branch yet, and it only rewrites page objects rather than core crawler control logic.

Even so, moving the repair loop into the spider simplifies our infrastructure while keeping a human in the loop for the hard problems. When a production spider is clearly broken, doing a speculative repair on the fly and letting a human review the code and data after the crawl finishes is often much better than halting delivery to wait for human approval.
Software that operates in unstable environments should be able to adapt while it runs. If you want to experiment with this approach or take the ideas further, you can check out the code at github.com/lopuhin/scrapy-phoenix.







