Spider monitoring made easy

A Scrapy job can exit cleanly after collecting zero products. It can also export 20,000 products with no prices. Neither failure needs to raise an exception in your spider, which means neither failure is visible if the only definition of success is an exit code of zero.

Spidermon gives a Scrapy project a place to define success in terms that matter to the data: how many items arrived, whether required fields are present, and what should happen when those expectations are not met.

A finished crawl tells you that the process stopped. A healthy crawl tells you that the data is still useful.

Part 1 put logging, pipelines, and feed exports in the right places. Part 2 moved page-specific extraction into typed Page Objects. This article closes the loop: it turns the resulting crawl statistics and item validation into monitors, then connects a failed monitor to an action that somebody can see.

What should a production spider prove?

Start with the failure modes of the specific spider, not a generic monitoring checklist. Our running catalog spider should not pass merely because it issued requests and wrote a file. It should show that it found enough products and that those products still meet the Product schema from Part 2.

The last column matters. Monitoring is not a promise that data is perfect. It is a way to turn specific, known bad states into visible events before downstream users discover them in a dashboard or a customer-facing report.

Enable Spidermon before writing a monitor

Install Spidermon and jsonschema, which its JSON Schema validation feature uses:

1pip install spidermon jsonschema

Enable the extension and tell it which monitor suite should run when the spider closes. Keep the configuration with the rest of your project settings, not hidden in a deployment script.

1# catalog/settings.py
2SPIDERMON_ENABLED = True
3
4EXTENSIONS = {
5    "spidermon.contrib.scrapy.extensions.Spidermon": 500,
6}
7
8SPIDERMON_SPIDER_CLOSE_MONITORS = (
9    "catalog.monitors.CatalogCloseMonitorSuite",
10)

Spidermon runs monitors in suites at defined points in the spider lifecycle. A spider-close suite is the natural first choice because the final Scrapy stats and the item-validation results are both available by then. The Spidermon getting-started guide documents the extension configuration and monitor lifecycle.

Do not begin by alerting on every retry or every 404. Start with the conditions under which you would not ship the resulting dataset.

Define a minimum item count that means something

The first monitor checks an intentionally modest, project-specific threshold. In a real catalog, derive it from the spider's normal output, its configured scope, and the consequences of partial data. A development crawl limited to one category should not be judged by the threshold of a nightly full crawl.

1# catalog/monitors.py
2from spidermon import Monitor, MonitorSuite, monitors
3
4
5@monitors.name("Catalog crawl")
6class CatalogStatsMonitor(Monitor):
7    @monitors.name("Minimum product count")
8    def test_minimum_product_count(self):
9        item_count = getattr(self.data.stats, "item_scraped_count", 0)
10        minimum_count = 100
11
12        self.assertGreaterEqual(
13            item_count,
14            minimum_count,
15            msg=(
16                f"Expected at least {minimum_count} products, "
17                f"but extracted {item_count}"
18            ),
19        )
20
21
22class CatalogCloseMonitorSuite(MonitorSuite):
23    monitors = [CatalogStatsMonitor]

Run the spider locally, then deliberately set minimum_count above the current output once to see the failure format in the logs:

1scrapy crawl products

That small exercise is worth doing. A monitor that has never failed in development is only assumed to be connected to your runtime.

Scrapy already collects the stats that make monitors possible. The Scrapy stats documentation is useful when you need to inspect the exact counter name produced by a downloader, middleware, pipeline, or your own component.

Validate items at the point where they leave the spider

Part 2 gave our Page Object a typed Product item. That prevents plenty of accidental field mistakes in Python, but the data still comes from the web. Schema validation gives the project an independent check on the item it is about to export.

Place the validation pipeline after the project's own cleanup pipeline. Validating before a pipeline normalizes a price or URL creates false alarms, while validating before a later pipeline changes the item creates false confidence.

1# catalog/settings.py
2ITEM_PIPELINES = {
3    "catalog.pipelines.ProductPipeline": 300,
4    "spidermon.contrib.scrapy.pipelines.ItemValidationPipeline": 800,
5}
6
7SPIDERMON_VALIDATION_SCHEMAS = ["schemas/product.json"]
8SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS = True

Here is a deliberately small schema. It requires all four fields from the Product item and rejects empty name and price values. Your schema should grow with your data contract, not with every possibility you can imagine on the first day.

1{
2  "$schema": "https://json-schema.org/draft/2020-12/schema",
3  "type": "object",
4  "properties": {
5    "name": {"type": "string", "minLength": 1},
6    "price": {"type": "string", "minLength": 1},
7    "description": {"type": "string"},
8    "url": {"type": "string", "format": "uri"}
9  },
10  "required": ["name", "price", "description", "url"]
11}

With SPIDERMON_VALIDATION_ADD_ERRORS_TO_ITEMS enabled, invalid items carry validation information in an _validation field. That is useful while diagnosing a new rule. For a production export, decide whether downstream systems should receive flagged records or whether the pipeline should drop invalid items using SPIDERMON_VALIDATION_DROP_ITEMS_WITH_ERRORS.

The Spidermon item-validation guide recommends making the validation pipeline the last one, and documents the schema, error-field, and item-dropping settings.

Fail the crawl when validation errors become data loss

Adding validation errors to items is observability. A monitor turns those errors into a decision.

1# catalog/monitors.py
2from spidermon import Monitor, monitors
3from spidermon.contrib.monitors.mixins import StatsMonitorMixin
4
5
6@monitors.name("Catalog data quality")
7class ProductValidationMonitor(Monitor, StatsMonitorMixin):
8    @monitors.name("No product validation errors")
9    def test_no_product_validation_errors(self):
10        error_count = getattr(
11            self.stats,
12            "spidermon/validation/fields/errors",
13            0,
14        )
15
16        self.assertEqual(
17            error_count,
18            0,
19            msg=f"Found {error_count} product field validation errors",
20        )

Add it to the suite from the first monitor:

1class CatalogCloseMonitorSuite(MonitorSuite):
2    monitors = [
3        CatalogStatsMonitor,
4        ProductValidationMonitor,
5    ]

This rule is intentionally strict. It fits a catalog where a product without a price is not acceptable output. Other use cases may tolerate a small number of missing optional descriptions, or may care about a percentage rather than an absolute count. Spidermon provides validation-monitor helpers for field-level counts and percentages when that is the more honest definition of failure. Do not set a tolerance just to silence a monitor. Explain what the tolerated records mean for the people using the data.

Use periodic monitors to stop bad long-running crawls early

Spider-close monitors are necessary, but they arrive too late for every problem. If a broad crawl has been returning error pages for 45 minutes, waiting for it to complete wastes capacity and leaves an operator with a larger incident to untangle.

Periodic monitors run a normal monitor suite at a configured interval while the spider is still running. They are a good fit for failures that become expensive as they continue: a rising error count, no increase in extracted items, or a job running past its intended duration. Keep final data-quality validation in the spider-close suite, where the complete output is available.

A periodic monitor is a circuit breaker for a crawl, not a replacement for its final quality gate.

For the catalog spider, close the crawl after 20 logged errors. The number is an example, not a threshold to copy. Choose one that reflects the target site's normal failure behavior and the point at which continuing is no longer useful.

1# catalog/actions.py
2from spidermon.core.actions import Action
3
4
5class CloseSpiderAction(Action):
6    def run_action(self):
7        spider = self.data["spider"]
8        spider.logger.error("Closing spider after a periodic monitor failure")
9        spider.crawler.engine.close_spider(
10            spider,
11            reason="closed_by_spidermon",
12        )

1# catalog/monitors.py
2from spidermon import Monitor, MonitorSuite, monitors
3
4from catalog.actions import CloseSpiderAction
5
6
7@monitors.name("Catalog periodic health")
8class CatalogPeriodicMonitor(Monitor):
9    @monitors.name("Error count remains below the limit")
10    def test_error_count(self):
11        error_count = self.data.stats.get("log_count/ERROR", 0)
12        maximum_errors = 20
13
14        self.assertLessEqual(
15            error_count,
16            maximum_errors,
17            msg=f"Crawl logged {error_count} errors, limit is {maximum_errors}",
18        )
19
20
21class CatalogPeriodicMonitorSuite(MonitorSuite):
22    monitors = [CatalogPeriodicMonitor]
23    monitors_failed_actions = [CloseSpiderAction]

Schedule the suite every 60 seconds:

1# catalog/settings.py
2SPIDERMON_PERIODIC_MONITORS = {
3    "catalog.monitors.CatalogPeriodicMonitorSuite": 60,
4}

Spidermon also provides built-in periodic monitors for execution time and item-count increase. Those are often a better first choice

A failed monitor should trigger an action

Logs are fine when you are running one spider in a terminal. They are not a monitoring strategy for scheduled jobs. Attach a failure action to the monitor suite so that a broken data contract reaches a channel where it can be triaged.

1# catalog/monitors.py
2from spidermon.contrib.actions.slack.notifiers import (
3    SendSlackMessageSpiderFinished,
4)
5
6
7class CatalogCloseMonitorSuite(MonitorSuite):
8    monitors = [
9        CatalogStatsMonitor,
10        ProductValidationMonitor,
11    ]
12    monitors_failed_actions = [
13        SendSlackMessageSpiderFinished,
14    ]

Keep credentials out of version control. For example, load the Slack configuration from the environment used to run the spider:

1# catalog/settings.py
2import os
3
4SPIDERMON_SLACK_SENDER_TOKEN = os.environ["SPIDERMON_SLACK_SENDER_TOKEN"]
5SPIDERMON_SLACK_SENDER_NAME = "catalog-monitor"
6SPIDERMON_SLACK_RECIPIENTS = ["#data-alerts"]

Spidermon also supports actions for email, Telegram, Discord, Sentry, reports, and custom integrations. Start with one destination that has an owner. Alerting an unmonitored channel is only a more elaborate way to write to a log. The actions documentation lists the built-in and custom-action options.

Choose the action by the failure

Do not make the first automated reaction "run the same spider again." Retrying can be sensible for a transient timeout. It cannot repair an HTML template that no longer has a .price element.

Keep the monitor suite small enough to trust

An early suite needs only a few checks that protect against known bad outcomes. For the catalog spider, start here:

1. At least the expected minimum number of products were extracted.
2. No required product fields failed schema validation.
3. A failed check sends one actionable notification to the owner.

Add checks when you can state the response to failure. For example, a sudden rise in retry count might warrant an alert only if it has a named owner and a clear threshold. Otherwise, it is a graph waiting to be ignored.

Zyte Web Data for Claude Code and Web Scraping Copilot 1.0 can speed up the work of creating spiders and Page Objects. The review loop is still yours. The same is true of the concerns raised in this practical review of AI-generated Scrapy projects: code that runs needs checks that say whether it produced usable data.

Questions developers ask about Spidermon

Should every validation error fail the crawl?

No. Fail on the errors that make the dataset unfit for its purpose. A missing product URL is usually serious. A missing optional marketing description may deserve a report and a follow-up instead. Make the distinction explicit in the schema and monitor threshold.

Can Spidermon replace tests for Page Objects?

No. Page Object fixture tests catch a regression against a known response before deployment. Spidermon checks the behavior and output of a real crawl. Use both, because they fail at different points in the workflow.

Should invalid items be dropped immediately?

Sometimes. During development, attaching _validation errors to items makes debugging easier. In production, dropping an invalid record may be safer than exporting it, provided the monitor still alerts you to the loss. Do not silently drop records and call the job successful.

Make a failure visible before the next scheduled crawl

Add one item-count monitor, one required-field check, and one failure action to an existing spider. Then break each condition on purpose in a safe environment and follow the signal from the Scrapy stats to the monitor result to the notification.

That is the beginning of operating a spider as a data system. Part 4 will return to acquisition: how to choose between Scrapy, APIs, and browser automation when the target site is more complicated than static HTML.