How can a data gathering project pass all your checks and yet still fail?
The usual reason - it’s incomplete. Sure, the rows that arrived may be fine, but the rows that never hit - well, they are missing. Nothing flags a row that does not exist.
We have seen customers sign off on datasets with thousands of items missing. They couldn’t tell, and neither could the checks, because everything that was present looked correct. As the DQ playbook explains, a successful extraction can still leave data incomplete or inaccurate.
This article is about that gap, and about what changes when an AI agent handles QA instead of a list of rules.
What good QA already covers
We have written about the foundations before, so here is a short recap.
- Define what “good” means before you automate anything. Item coverage, field coverage and field accuracy are the three questions to start from. (AI won’t fix your data quality)
- Write the expectations down as schemas, track coverage against a baseline, and compare each delivery with the last one. (How Zyte’s extraction experts guarantee data quality)
- Use a model to judge whether individual fields look real, and stop a crawl when too many do not. (Stopping a crawl when the data stops looking real)
For a practical Scrapy example, see how to monitor completeness, correctness, consistency and volume with Spidermon and Claude Code.
All of this works. Each check still answers a question someone thought of in advance.
The question rules can’t answer
The question a customer cares about is simple:
Did we collect what we promised?
That is a judgment, not a rule. It depends on:
- What was agreed.
- How the source behaves.
- What the run produced.
A data schema can tell you a price is a number, but it can’t tell you, say, that a whole category of products never got visited.
Rule-based data quality assurance (QA) handles the failures you anticipated. But web data also fails in ways nobody anticipated. For example:
- A section of a site changes layout.
- A filter quietly narrows the results.
Each of those cases needs its own rule, and you usually write the rule after it has already cost you a delivery.
What an agent changes
An agentic approach treats QA as a question to investigate rather than a checklist to run. In practice that means three things:
- Plain English: It works from the same human linguistic agreements that people use. Project scope is rarely written as a schema. It lives in agreements, tickets and conversations, and an agent can read those and compare them with what was delivered.
- Human behaviour: It investigates the way a person would. It can look at the source, compare it with the output, and keep following a doubt until it is resolved instead of stopping at the first passing check.
- Probabilistic investigation: It notices things no-one wrote a rule for.
An example
In one case I worked on, the problem showed up in the timestamps on the extracted items.
None of the project’s usual deterministic checks looked at them.
But the agent read the output, saw that the timestamps did not fit how the source behaves, and concluded the scraper was collecting the wrong thing.
A careful person might have caught the same thing, but only by looking at that project, on that day, with enough time to read the output closely.
By contrast, an agent can do that kind of reading on every delivery.
Where it falls short
It’s not perfect, of course.
- It is only as good as what you agreed. If the scope was never written down, the agent has nothing to compare against, which is the same lesson as before.
- Judgment calls stay with people. Whether a gap matters to a given customer is a business decision.
- An agent can be confidently wrong, like any automated check. For broader context, see why AI agents struggle with web scraping. Independent human sampling is how you find out how often.
What to change in your own QA
What does this tell us about how you can assure data quality in the agentic data gathering era?
- Write acceptance criteria as statements you can test, not as intentions in a kickoff email.
- Measure coverage as well as validity. A clean delivery that is short is still a failed delivery.
- Before every sign-off, ask what would be missing and how you would know.
- Give whatever does your QA, human or automated, access to what was promised and not only to the data.
- Sample independently against whatever your automation reports.
For Scrapy teams, Scrapy Cloud’s built-in spider monitoring and data QA tools can help put those checks into practice.
Schemas, thresholds and spot checks tell you whether the data you have is good. An agent can help answer whether you have the data you were supposed to have.