Self-healing spiders: letting an agent fix your Scrapy crawl, with a lid on it

Would you let an agent fix your production spider at 3am, with nobody watching? That was the question I opened my workshop with at Extract Summit Austin in October 2026, and the honest answer depends far less on the model than on everything you build around it: what notices the problem, what the agent is allowed to touch, what decides whether the fix is any good, and when a human gets woken up instead.

This article walks through the two demos from that workshop, with the numbers we measured while building them, the mistakes that taught us the most, and a companion repository on GitHub so you can run both yourself.

What we mean by a self-healing scraper

A self-healing scraper notices that its data went wrong, fixes itself within limits you set, or hands over to a person. That definition is deliberately narrow, because "scraper breaks" covers three very different failures and only one of them is a good fit for an agent today.

The first is the fragile selector: a redesign moves the class names, the page still contains the data, and the spider simply cannot find it any more. An agent can read the new markup and repair that. The second is JavaScript rendering, where the HTML arrives empty, and here the data is often already sitting in the page as inline JSON, so the fix is to parse that rather than to reach for a browser (although sometimes you do need a browser). The third is blocking: a bot wall sends a challenge page instead of the markup, and no selector in the world heals that, so you fix access first, for example with Zyte API unblocking. Both demos in the workshop heal the first kind.

The agent proposes and the script decides

The design principle that everything else hangs off is a split between two kinds of parts. The deterministic parts are plain code: the Scrapy spider, the checks, an answer key, a guard that reverts edits to locked files, an attempt cap, a budget, and the step that re-runs the crawl and judges the result. The non-deterministic part is the language model, which reads the broken output and the page, edits one file, takes a different path every run, and whose claim that "it works" counts for nothing.

The model lives in one box with a lid on it. If you have read the earlier article on loop engineering, this is the same idea applied to a spider: the prompt is a single component, and what makes the system safe is the goal, the test, the tools, and the budget around it.

Demo one: the redesign breaks the spider overnight

The first demo, in the brittle-selector-healing folder of the repo, runs entirely offline. A spider scrapes a bookshop using three selectors for three fields, written against site/v1.html:

1def parse(self, response):
2    for book in response.css("article.product_pod"):
3        price = book.css("p.price_color::text").get()
4        yield {
5            "title": book.css("h3.title::text").get(),
6            "price": float(price.strip("£")) if price else None,
7            "availability": book.css("p.availability::text").get(),
8        }

Run against the first version of the page, item_scraped_count is 10 and four checks go green: the item count, that every field is present, that values are sane, and a golden sample. The golden sample is an answer key of three books whose price and stock status were copied by hand from the live site, and it turns out to be the most important check of the four.

Then the bookshop redesigns. The same 10 books are on the page with the same prices, but every class name has changed and there is a new struck-through "was" price next to the current one. Running the same command produces this:

1'item_scraped_count': 10,
2{"title":null,"price":null,"availability":null}

Nothing crashed, the log looks like last night's, and every field is empty, which is the failure mode that makes selector breakage so expensive in production, because a crawl that errors gets noticed and a crawl that quietly succeeds with nulls does not.

The heal loop is a shell script around one agent call

heal.sh runs Claude Code in headless mode with claude -p, one prompt in, one session, then it exits. If you want the background on running an agent this way, the headless mode article in our harness engineering series covers it in detail. The part that matters here is the options:

1# simplified from heal.sh
2opts=(--permission-mode bypassPermissions --tools "Read,Grep,Glob,Bash" --strict-mcp-config)
3claude -p "$prompt" --output-format stream-json --verbose "${opts[@]}"

Because nobody is there to click "allow" in an unattended run, bypassPermissions means nothing will ask, which is why the tool list has to be tight. --tools is an allowlist of the only tools the agent gets, and --strict-mcp-config stops a headless run from loading every MCP server configured on the laptop. The prompt tells the agent which command must pass, pastes in the failing output, says to edit only books.py, and forbids touching check.py or anything under site/. A guard then copies those files before the first attempt and reverts them if an edit lands, failing that attempt.

After each attempt the script, not the agent, re-runs the crawl and the checks. Across seven heals of the same break with Claude Opus 5.5 on September 28 and 29, 2026, the agent fixed the spider on attempt one every time, at $0.07 to $0.16, in 24 to 31 seconds and four to seven agent turns. In the three heals where we looked, the agent never ran the check itself, so the script's re-check did all of the proving. Those figures describe seven samples of one break, and the slide that introduced them said so plainly: your mileage will vary with the model, the effort setting, the tool access, the page complexity, and both prompts.

A green check is not a correct fix

I then showed a second fix, saved in the repo as books_wrong.py, whose output had every field filled in with a plausible title, price, and stock status, and I asked whether they would ship it. Three of the four checks passed.

The wrong fix took the first price on each card, which on the redesigned page is the struck-through old price, so every price was 25% too high while still being a number between 1 and 200. Only the golden sample, which asks whether A Light in the Attic actually costs £51.77, caught it. A fix written by a rushed human or a weaker agent looks exactly like this one, so the checks have to compare values against answers you know, because a filled-in field proves nothing.

Tool access decides whether the agent can heal anything

In the live run, I had quietly removed the shell from the agent's tool list, leaving only Read, Grep, and Glob. The agent found the correct fix and had no way to write it. Over three runs it escalated every time, after two failed attempts, at $0.11 to $0.24, and in every run the ESCALATION.md it wrote said that it lacked an edit or shell tool. Restoring the shell tool healed the spider on attempt one in all three runs at $0.07 to $0.12.

The heal with no write access still cost real money and fixed nothing, which is the case for a budget and for escalating early. When we pointed the loop at a page where the prices sit behind a sign-in, so there was nothing to fix, it escalated in all five of our rehearsal runs (two with two attempts, three with three), never invented a price, and named the cause in its own message.

Four lessons that cost us a run to learn

Several of the findings from building this were things I had not expected, and the first of them changed how the script is written.

1. Use an allowlist, not a denylist. We first denied Bash, Edit, and Write. In one of five runs the agent used ToolSearch to discover Monitor, a tool that runs shell commands, and wrote books.py with it anyway. Passing the exact tools you want with --tools fixed that.
2. Isolate the agent from your own setup. A headless run still loaded every MCP server configured on the laptop until we added --strict-mcp-config, and it also reads every CLAUDE.md above the folder it runs in, so rehearse from a clean directory such as /tmp.
3. Prompt for correct, not for green. Saying what correct data looks like, and that giving up is acceptable, is safer than "make the check pass", which invites a fake.
4. Escalate after two tries. In every failed run the agent's own message named the cause, so a second attempt rarely bought anything a human reading the message would not have seen immediately.

Demo two: heal the crawl while it is still running

Demo one waits for the crawl to finish, which is fine for 10 books and wrong for a crawl that runs for hours. When a selector breaks at hour three you have three options, and each one costs something: keep going and pay for every page downloaded and stored broken, stop and let the data arrive late once someone notices at 3am, or restart and re-fetch three hours of pages. The fourth option is to pause, fix, and carry on.

Spidermon detects, and the Scrapy MCP is the agent's window

Spidermon already checks a crawl while it runs, and our earlier piece on spider monitoring covers the basics. What was missing was a way for an agent to look inside the live process, which is what the Scrapy MCP server provides. It attaches through the remote control extension that Scrapy 2.19 turns on by default, and exposes four tools: list_jobs and status find the running crawl and read its stats, execute runs Python inside the crawl process, and inspection_reference is the server's own guide to what is safe to poke at. We cover how that extension works in Inside Scrapy's RemoteControl extension, and the announcement in Scrapy MCP: debug your crawl while it runs.

The project, in the scrapy-mcp-agent-mid-fix-crawl folder of the repo, crawls books.toscrape.com with scrapy-poet page objects, a pattern we cover in building maintainable spiders with scrapy-poet, and that matters because a fix can then be one field on one page object instead of a change to the spider. The workshop-specific part is a small list of the last 10 product responses, kept in memory as crawler.recent_responses, which lets the agent test a candidate selector against pages the crawl has already downloaded, with no new request and no restart.

Watch, pause, fix, resume

A Spidermon periodic monitor checks the items scraped since its last check, twice a second, and the action it triggers pauses the crawl engine instead of stopping it:

1SPIDERMON_PERIODIC_MONITORS = {"base.quality.Tripwire": 0.5}
2
3class Tripwire(MonitorSuite):
4    monitors = [LiveQuality]
5    monitors_failed_actions = [PauseForHealing]

Pausing is a freeze-frame: the queue and the stats are kept, in-flight pages land, and the process stays alive while the agent works, which is what buys it time. When the agent finishes, the script runs the fixture tests, and only if they pass does it load the fixed page objects into the live crawl and resume it.

The agent's reach is narrow by construction. It gets Read, Edit, and the four Scrapy MCP tools, with no Bash, no Write, and no other MCP server, and its prompt says to edit one file only and not to pause, unpause, stop, or change the crawl. Pause, unpause, and load belong to the script, so the agent can look at the crawl and edit one file but cannot steer it.

What the live run looks like

We broke the price selector on purpose, because we cannot redesign someone else's site, and said so on stage. Without the agent, the crawl finishes with 200 books and no prices. With it switched on, this is a trimmed recording of a Sonnet run from October 3, 2026:

1✗ Spidermon: 8 of 8 new items missing fields
2⏸ crawl paused · 16 books so far
3━━ attempt 1/2 ━━ agent attaching to the crawl through the Scrapy MCP…
4→ mcp execute … ← …/set-me-free_988/ None £17.46
5→ Edit base/pages/books_toscrape_com.py
6attempt 1 PASS · $0.06 · 10s · 6 turns
7▶ crawl resumed with the fix, no restart

The agent's whole change was one line, swapping the dead selector for the one that matches the page:

1- raw = self.css("div.product_main p.price-now::text").get()
2+ raw = self.css("div.product_main p.price_color::text").get()

Across nine runs on 200 books, three each on Haiku, Sonnet, and Opus, the crawl never restarted and between eight and 16 books came out without a price. Those were already downloading when Spidermon paused the crawl, and they stay broken in books.jsonl because the loop does not re-crawl them, which is a trade-off to decide on deliberately rather than discover later.

Which model should heal

We ran the same break through three models, three runs each, on October 1, 2026:

All three fixed it, so for a one-selector break the difference is time and money rather than success, and a different model is a different agent, so re-measure before you switch.

Two bugs from the build that shaped the design

The first bug is that a tripwire counting every item so far re-paused the crawl right after a fix, because the broken books were still in the total. The monitor now judges only items scraped since its last check. The second is that Haiku re-imported the page object module inside the crawl process, so the tests passed while the live crawl stayed broken, since the crawl held the original classes. The fix now goes onto the classes the crawl is actually using.

Two design decisions followed from these. The fixture tests, saved with scrapy savefixture, are the answer key, and the agent cannot run them because the script does. And the agent swaps one field on a page object rather than patching the spider, partly because the MCP's own reference describes patching a live crawl as "a separate, riskier mode".

Designing your own loop

Both demos are the same five steps wrapped around different parts. Detection is check.py in the first and Spidermon in the second, context is the prompt plus the failing output, the action is claude -p behind an allowlist, verification is the script's re-check in the first and the fixture tests in the second, and the decision is pass and ship or fail and retry, with ESCALATION.md and a cap of two attempts or a fixed spend at the end. The question I left the room with is the one to take back to your own pipeline: what detects it, what may the agent touch, what decides, who gets paged, and what is the budget?

Everything above lives in one repository, zytelabs/self-healing-spiders, which you can clone and run, with each demo in its own folder and a README listing every command. You need uv and a signed-in Claude Code CLI, and each heal costs real tokens, so the READMEs list what we measured. Run the demos from a copy outside any folder that has its own CLAUDE.md, or the agent will read those instructions as well. For more from the event, see our five takeaways from Extract Summit 2026 Austin, and if the selector fix is solved but your pages never arrive, that is a job for Zyte API, which you can try for free.