A scraper can finish its crawl with every request returning a 200, a full set of items in the output and nothing in the price column, and you find out when someone opens the export. The site changed its layout and renamed a CSS class, the selector stopped matching, and because a missing value isn't an error, nothing in the crawl complained. That gap is the problem self-healing scrapers are meant to close, and it's the kind of scraper maintenance that usually eats an afternoon every time a site changes its markup.
I've been building a small self-healing agent that catches this while the crawl is still running and attempts a repair. When I change the price class on my own test shop, the monitor stops the crawl, the agent works on a copy of the project, captures the changed page, updates the selector and runs the tests, and in my demo it took about a minute from the start of the crawl to a repaired file. That was one failure on one page of a test shop, so it's an illustration and not a benchmark. What I took from building it is that the agent is the smallest part. Most of the code copies the project, runs subprocesses, validates results and moves files back, while the model reads some files, edits two of them and runs the tests, so what decides whether a repair is any good is what you've already built around it.
I wrote up the build itself separately on dev.to, and Ayan covered the two self-healing demos from our Extract Summit Austin workshop, including healing a live crawl through the Scrapy MCP, in Self-healing spiders: letting an agent fix your Scrapy crawl. This post covers the foundations a self-healing web scraper needs, in the order I'd put them in place. It's about repairing extraction, meaning the selectors and page objects that read the data. Blocks, burned sessions and changed navigation fail differently and have different owners, so I've left them out.
Detect broken selectors when a website changes, during the crawl
If you only learn about a missing price from the finished export, the damage is done and every fix after that costs a re-crawl. In my project every product passes through a pipeline that records missing prices in Scrapy's stats, and after every tenth product a Spidermon monitor suite runs against those stats and asks the spider to close if any price is missing or blank. If you haven't used Spidermon before, I walked through setting it up in Spider monitoring made easy.
The harder part is deciding what counts as a failure, and that has to come from you before anything else does. A missing price stops my crawl because that's the right rule for this shop, but on another site an unavailable product might legitimately have no price, and an agent handed a wrong rule will try to repair something that was never broken. Write the rule down, make it cheap to check on every item, and make it fire early, because the status codes will look fine the whole time, and that's the difference between scraper monitoring that watches responses and data quality monitoring that watches the fields your CSS selectors return. A present value can still be wrong, and if you want a check on whether fields still look real and not just whether they exist, scrapy-jev does that with a fast AI model.
Hand the agent a specific failure report
When the monitor closes the spider, it also writes a structured event with the reason, the field, one offending URL and the page object class responsible for extracting it:
1{
2 "reason": "missing_product_price",
3 "field": "price",
4 "url": "https://coffee.hylnd7.com/labs/classic/product/midland-papua-new-guinea-bea016",
5 "page_object": "base.pages.coffee_hylnd7_com.CoffeeProductPage"
6}I keep the first event and ignore later ones, because Scrapy is asynchronous and requests already in flight can finish during shutdown, so the last response you see isn't necessarily the first one that failed. The agent starts with a field, a URL and a class to look at, and never has to find the problem in a crawl log. A vague failure tends to produce a vague repair, which is why this report matters more than it looks.
Make the extraction code easy to target
If navigation, extraction, validation and storage all sit in one spider, a repair agent has a lot of code it could change and no good reason to change only one part. I use scrapy-poet to move extraction into page objects, so the spider crawls and a page object extracts the fields for one kind of page, and there's a fuller walkthrough of that layout in Building maintainable spiders with scrapy-poet. The failure says price, and a price() method on the page object controls how that value is extracted, which gives the agent, and the person reviewing its work, one place to look.
That structure also lets me limit what the agent can touch. Its write tool can only edit the page-object file and the new fixture's expected output, it has no shell, and none of its tools can reach the live project, the settings or the monitors. The scope is still the file and not the method, which turned out to matter, because currency() and currency_raw() read the same element as price() and the agent fixed all three in one pass. The temporary copy isn't a security sandbox either, since page objects and pytest execute Python on the host, so this is a local tool for a project I control.
If you're starting a project and want this layout from the beginning, Zyte's Agent Skills generate it for a coding agent, a Scrapy project with web-poet page objects, pytest fixtures and a spider.
If you'd rather watch the build than read about it, this is the walkthrough I recorded of the project, including the repair running against a deliberately changed price class.
Build scraper tests that can fail
The tests are what turn an edit into a repair, and they're the part of self-healing web scraping I'd least want to skip. The wrapper copies the project into a temporary directory, runs scrapy savefixture there to capture the changed page through the project's own downloader configuration, and gives the agent that fresh fixture to work from, so it develops selectors against the real response and not a guess at it. The new fixture becomes a regression test for the changed markup, and the existing fixtures check that the old layout still works, which matters because the repair added the new class next to the original selector and not in place of it.
I don't take the agent's word that the tests pass. The controller runs everything again itself, checks that protected files haven't changed and that nobody edited the live project during the repair, and only then copies back the selector file and the new fixture, with a backup and a diff. A separate guard requires the extracted price to be a nonblank, finite, nonnegative decimal that matches the fixture, which stops the agent writing null into the expected output and passing a comparison while leaving the problem in place.
These checks have a gap I'd rather say out loud. The agent also fills in the new fixture's expected output, so they show that the selector and the fixture agree with each other and not that the price is right, and a nonblank price can still be an instalment amount, an old price or the wrong product's. The answer is to heal back to something known, such as the last good value for the same page or the value as visible text in the raw HTML, and until the checks do that, reviewing the diff and the source page is still worth doing.
Four questions to ask before building self-healing scrapers
- Would you find out about a missing field during the crawl, and have you decided what counts as one?
- When it fails, can you point at one field, one URL and one page object without reading a log?
- Does extraction live in one place an agent could be allowed to edit, with everything else out of its reach?
- Do you have tests that would fail on a wrong repair, and something known to compare the new answer against?
If any answer is no, that's where I'd put the work before adding a model. If they're all yes, the agent's job is small enough to review, which is most of the point.
Questions and answers
What is a self-healing scraper?
A self-healing scraper notices when its own extraction has stopped working, usually because a site changed its markup, and repairs the code that reads the data without someone having to find the problem first. In my version, a monitor spots the missing field during the crawl, and an AI agent updates the selector on a copy of the project and has to pass the tests before anything goes back into the live code.
Why do web scrapers break when a website changes?
Most scrapers find data with CSS selectors or XPath tied to class names and the position of elements on the page, so a redesign, a renamed class or a moved element leaves the selector pointing at nothing. The request still succeeds and the page still loads, which is why a broken scraper often produces empty fields and no errors at all, and why you need a check on the data rather than on the responses. If you're tracking down which selector broke, browser DevTools is the quickest way to compare the old markup with the new.
How do you maintain web scrapers at scale?
Keep extraction in page objects so each fix has one place to go, save real pages as test fixtures, and monitor the fields your scrapers return during every crawl, not just the status codes. Once those are in place, a self-healing agent can take on the routine selector fixes, but it's the last step and not the first, because without the other three you can't tell whether its repairs are right.
Can an AI agent fix a broken web scraper on its own?
It can make the edit, and for a renamed class or a moved element an LLM-based agent often makes the right one, but I wouldn't let it decide on its own that the repair is correct. The agent writes both the selector and the expected output for the new fixture, so the tests only show that the two agree, which is why I still review the diff and compare the new value against the source page.
Does self-healing web scraping help with blocks and bans?
Not the kind I've described here, which only repairs extraction. A 403, a challenge page or a burned session isn't a selector problem, so editing a page object won't fix it, and those failures are better handled by whatever manages your access to the site, whether that's your own proxy and browser setup or a service like Zyte API.
Do I need Scrapy to build a self-healing scraper?
No, but you need the same pieces in whatever you use: a check that runs during the crawl, a failure report that names the field and the page, extraction code kept apart from everything else, and tests built from saved pages. Scrapy, Spidermon and scrapy-poet happen to give you all four, which is why I built it that way.
Is it safe to let an agent edit my scraper code?
Only with limits. My agent can edit one page-object file and one fixture, it has no shell, and it works on a temporary copy that only gets copied back after the controller reruns the tests itself. That copy isn't a security sandbox, though, because the tests run Python on the host, so I'd only run this on a project and a machine I control.
