Building maintainable spiders with scrapy-poet

A Scrapy callback can begin life as six lines of selectors and end up responsible for following links, interpreting a page, normalizing an item, deciding whether data is missing, and fetching a second endpoint when the price is loaded separately. It still runs, so it is easy to call that design successful.

Until the site changes. Then the code that needs changing is also the code that decides how to crawl, and a selector fix becomes a risky edit to the spider's control flow.

scrapy-poet is valuable because it gives extraction logic its own home. It integrates web-poet, a Page Object framework, with Scrapy so that a spider can concentrate on crawling while Page Objects describe how to turn a particular kind of page into a typed item.

This is not an argument that every 20-line spider needs another abstraction. It is an argument that the callback is the wrong long-term home for reusable extraction logic.

The callback starts simple, then quietly owns too much

Here is the product callback from Part 1 of this series, expanded with a description and a JSON-LD fallback for a price. Nothing here is outrageous. That is why this pattern lasts so long.

1import json
2
3
4def parse_product(self, response):
5    name = response.css("h1::text").get(default="").strip()
6    price = response.css(".price::text").get(default="").strip()
7
8    if not price:
9        structured_data = response.css(
10            "script[type='application/ld+json']::text"
11        ).get()
12        if structured_data:
13            data = json.loads(structured_data)
14            price = data.get("offers", {}).get("price", "")
15
16    yield {
17        "name": name,
18        "price": price,
19        "description": " ".join(response.css(".description *::text").getall()).strip(),
20        "url": response.url,
21    }

Now imagine a second spider needs the same product interpretation, or that the catalog moves to a new domain with different selectors but the same output schema. Copying the callback is fast. Keeping copies aligned is where the cost arrives.

The boundary we want is straightforward:

• The spider owns where to go and which links to follow.
• A Page Object owns what one page means and how to extract it.
• A pipeline still owns rules applied to every yielded item, such as the price validation from Part 1.

That division keeps an ordinary selector change out of the crawling strategy. It also makes the right unit of reuse obvious: the page, not a large utility function passed a response, a spider, and a few flags.

Put product extraction and its output contract in a Page Object

A web-poet Page Object receives page inputs, such as the downloaded response, and exposes an item. Use an attrs class instead of a dictionary for that item. It makes a misspelled field or a missing required value fail close to the extractor, rather than silently creating a new key in a result file.

Returns[Product] declares the item contract for the Page Object. Combined with @field, it lets web-poet build the Product instance automatically, so there is no hand-written to_item() dictionary to keep in sync with the model.

1# catalog/items.py
2from attrs import define
3
4
5@define
6class Product:
7    name: str
8    price: str
9    description: str
10    url: str

1# catalog/pages/products.py
2import json
3
4from web_poet import Returns, WebPage, field, handle_urls
5
6from catalog.items import Product
7
8
9@handle_urls("example.com")
10class ProductPage(WebPage, Returns[Product]):
11    @field
12    def name(self) -> str:
13        return self.css("h1::text").get(default="").strip()
14
15    @field
16    def price(self) -> str:
17        visible_price = self.css(".price::text").get(default="").strip()
18        if visible_price:
19            return visible_price
20
21        structured_data = self.css(
22            "script[type='application/ld+json']::text"
23        ).get()
24        if not structured_data:
25            return ""
26
27        data = json.loads(structured_data)
28        return data.get("offers", {}).get("price", "")
29
30    @field
31    def description(self) -> str:
32        return " ".join(self.css(".description *::text").getall()).strip()
33
34    @field
35    def url(self) -> str:
36        return self.response.url

The Page Object is not magically more correct than the callback. Its advantage is that the responsibilities and output schema are visible. A reviewer looking at ProductPage can discuss selector behavior and the Product contract without first understanding pagination, request metadata, or retry policy.

Keep the item class deliberately dull. Reuse Product across sites that produce the same schema, and keep site-specific parsing and cleanup in Page Object fields. A typed item is a contract, not a second place to hide extraction logic.

This is also a useful place to be honest about the limitations. A page with invalid JSON-LD will raise JSONDecodeError in the example above. Whether that should fail the item, produce a warning, or fall back to a different source is a data-quality decision. Do not swallow it just to make the crawl look healthier. Part 3 will deal with making those failures observable.

The scrapy-poet basic tutorial shows the same separation with a smaller example. The web-poet items guide explains its recommendation to use item classes, while the Page Object documentation goes deeper into available page inputs.

Enable scrapy-poet as a Scrapy add-on

For Scrapy 2.10 and later, scrapy-poet is enabled through Scrapy's add-on system. Add-ons are designed to package related Scrapy component configuration, rather than asking every project to manually copy middleware and fingerprinter settings. Scrapy's add-on documentation explains how their priorities and overrides work.

1pip install scrapy-poet

1# catalog/settings.py
2ADDONS = {
3    "scrapy_poet.Addon": 300,
4}
5
6SCRAPY_POET_DISCOVER = ["catalog.pages"]

SCRAPY_POET_DISCOVER tells the integration where to find Page Objects that have URL rules. Even if your first Page Object is imported directly by a spider, set this up now. It makes the project layout predictable once you add site-specific Page Objects. The setup guide documents both settings and the configuration required for older Scrapy versions.

Ask scrapy-poet for the item the spider needs

With a URL rule and Returns[Product] in place, a spider callback can ask for Product directly. scrapy-poet chooses the Page Object that can produce that item for the requested URL, builds it, and injects the completed item into the callback. The callback no longer needs the response for extraction.

1# catalog/spiders/products.py
2import scrapy
3
4from catalog.items import Product
5from scrapy_poet import DummyResponse
6
7
8class ProductsSpider(scrapy.Spider):
9    name = "products"
10    allowed_domains = ["example.com"]
11    start_urls = ["https://example.com/catalog"]
12
13    def parse(self, response):
14        links = response.css(".product-card a::attr(href)").getall()
15        yield from response.follow_all(links, callback=self.parse_product)
16
17    def parse_product(self, response: DummyResponse, product: Product):
18        yield product

DummyResponse makes the dependency explicit: this callback consumes the item, not Scrapy's response object. If you need the Page Object itself, perhaps because the callback combines it with crawling logic, request the Page Object and await page.to_item() instead.

For the common case where a product page produces one item, callback_for() removes the callback entirely:

1import scrapy
2
3from catalog.pages.products import ProductPage
4from scrapy_poet import callback_for
5
6
7class ProductsSpider(scrapy.Spider):
8    name = "products"
9    allowed_domains = ["example.com"]
10    start_urls = ["https://example.com/catalog"]
11
12    parse_product = callback_for(ProductPage)
13
14    def parse(self, response):
15        links = response.css(".product-card a::attr(href)").getall()
16        yield from response.follow_all(links, callback=self.parse_product)

Store callback_for(ProductPage) on the spider rather than creating it inline in response.follow_all(). The scrapy-poet documentation notes that the inline form does not work with Scrapy disk queues. That is exactly the kind of implementation detail you would rather learn before a crawl needs persistent scheduling.

Use URL rules when one spider needs more than one page implementation

The Page Object above describes one site's product pages. Reuse becomes more interesting when the crawl strategy stays the same while the page markup changes by domain.

web-poet can select a Page Object implementation using URL rules. Annotate a concrete class with handle_urls(), keep the common Product contract in a base class, and let discovery register the rule.

1# catalog/pages/example_store.py
2from web_poet import handle_urls
3
4from catalog.pages.products import ProductPage
5
6
7@handle_urls("example-store.com", instead_of=ProductPage)
8class ExampleStoreProductPage(ProductPage):
9    @field
10    def name(self) -> str:
11        return self.css("h1.product-title::text").get(default="").strip()

This does not mean every multi-site project should become a generic crawler. Different sites often have genuinely different navigation, login, pagination, or data contracts. Share the crawler only where the strategy is actually the same, and keep site-specific work explicit where it is not.

The payoff is most visible when it is time to replace a selector. You edit the relevant Page Object, test it against its fixture, and leave the rest of the crawl alone. The rules section of the scrapy-poet tutorial covers the fuller pattern, including explicit overrides.

Test an extraction rule without running a whole crawl

Most spider tests either mock so much that they no longer resemble a page, or run a real crawl and wait for the network. Page Objects give you a better middle ground: save the inputs and expected output for a representative page, then test the parser locally.

scrapy-poet supplies savefixture to capture a Page Object's dependencies and its to_item() result. The generated fixture can then be discovered by the pytest plugin that comes with web-poet.

1scrapy savefixture catalog.pages.products.ProductPage \
2  "https://example.com/products/widget-42"
3
4python -m pytest

The fixture becomes a regression test. When a selector change causes a price or name to move, the test can fail before a scheduled crawl produces a quiet column of empty values. The scrapy-poet testing guide and web-poet test documentation explain the fixture layout, expected output, and how individual fields are tested.

Fixtures are not a substitute for live checks. They tell you whether your extraction code still interprets a known response as expected. They cannot tell you that the site has changed, that a login page arrived instead, or that the listing page stopped linking to products. You need both fixture tests and a small, monitored live crawl.

When scrapy-poet is the wrong extension point

Use scrapy-poet when the problem is page interpretation, reusable extraction, or dependencies needed to interpret a page. Do not use it as a container for every Scrapy customization.

• Use an item pipeline for project-wide item validation and persistence.
• Use downloader middleware for request and response behavior that applies before the spider sees a page.
• Use a Scrapy extension and signals for crawl-wide behavior, such as reporting or metrics.
• Keep code in the spider when it is navigation strategy that only that spider owns.

This matters because a framework can make bad boundaries easier to decorate. A Page Object that schedules all requests, sends Slack messages, and writes to a database is still a callback with a nicer name.

If you are using AI-assisted tooling to create Page Objects, keep the review focused on the contract: which URL does this class handle, which inputs does it need, and exactly which item does it promise to produce? Zyte Web Data for Claude Code and Web Scraping Copilot 1.0 both make Page Objects part of the development workflow. The gaps between runnable code and a shippable Scrapy project are also the subject of this recent practical review. None of these tools remove the need to decide where project behavior belongs.

Make one callback smaller this week

Pick the callback that makes you hesitate before editing it. Move only the page-specific extraction into a Page Object, leave crawling and pipelines where they are, and generate one fixture for the page that has caused trouble before.

If that leaves the spider with a clear description of how it moves through the site, you have found a useful boundary. In Part 3, we will use Spidermon to define what a healthy crawl looks like and act when the spider stops meeting that definition.