4 key steps to develop an Automated Data QA process

Much is said about quality assurance and the automated data QA process. But do you really know how to get around doing it in the right way?

Developing an automated data QA process is easier said than done.

It's essential to help you obtain accurate data, but if not done appropriately, it can end up firing back on all your data collection (and business) efforts.

For example, working with inaccurate data will waste time and internal resources, as you scramble trying to properly understand a problem. Most likely leading to the wrong conclusions and negatively impacting your project or company's operations.

As a result, major operational issues may arise, such as losing customers and a decline in revenue. Therefore, it's important you ensure the data you're using for your project is of the highest quality possible.

This is where quality assurance comes into play.

To make sure you have reliable and high quality data, we recommend developing an automated data QA process made up of four layers.

Pipelines
Spidermon
Manually-Executed Automated QA
Manual/Visual QA

If you're looking to improve your web data quality assurance process, you need to start by developing an internal process made up of a four-layer methodology that communicates with all your system procedures.

This article will walk you through and clearly detail the four layers of the automated data QA process.

Streamline automated data QA

When looking to streamline your automated data quality assurance process, it's more important than ever to ensure that the data obtained is accurate and reliable.

Unfortunately, this isn't always the case, which is why it's important to have a clear plan in place for your automated data QA process.

As already mentioned, at Zyte we recommend that you apply a similar four-layer QA process, since the data quality assurance process we apply to all the projects we undertake with clients.

Create a pipeline that systematically tests each stage of data acquisition and processing.

Use spidermon to automate the manual process of data analysis.

Establish a vision-based quality assurance process to ensure accurate results.

Ensure your process is reliable and error-free by implementing manual vision quality assurance.

So now, let's go into further detail for each of the layers involved.

Validate data with Scrapy constructs

Pipelines are rule-based Scrapy constructs designed to cleanse and validate the data as it is being scraped. In order to ensure that the data you're using is of the highest quality is through the use of pipelines in Scrapy.

Typically, they include a number of rules (e.g. encoded in JSONSchema) that the data must adhere to, in order for it to be considered valid.

By setting up a pipeline, you can automatically check your data for errors and correct them as they occur, ensuring that only clean, accurate data makes it into your final dataset.

This saves you the time and effort of having to check your data manually. In addition, it also provides assurance knowing that your dataset is always of the highest quality. Implementing an automated data QA process using Scrapy pipeline is a key step to to improve the quality of your data.

One of the benefits of starting your Scrapy project by incorporating a pipeline, is that it can help to ensure that the data being scraped is of the highest quality possible.

For example, you can set a rule in the pipeline that defines all name tags must be at least three characters long.

So, if a product is scraped with only one or two characters long, it will be dropped from the dataset as it will be considered invalid.

The Spidermon framework

Spidermon is a spider monitoring framework we’ve developed to monitor and validate the data as it is being scraped.

You have probably already been in the following situation.

You're in the middle of a project, data is flowing in from various sources, and suddenly something goes wrong. A key piece of data is missing, or worse, it's inaccurate. Suddenly your project is stuck, while you try to track down the source of the problem and just can't figure it out.

Or worse, towards the end of your web data scraping process, while you are waiting for a large dataset to finish scraping, you find out that there was an error in the process and have to start all over again.

It's frustrating, time-consuming, and can cause a lot of headaches.

So…Spidermon to the rescue.

Spidermon benefits:

Anticipate errors and catch them early on

Identifies any errors that occur during the process and sends alert messages

Saves you time with having to start over

Helps ensure high quality data is extracted

The biggest advantage of using Spidermon is that it can catch most errors early on, before any major problems arise. It can be configured to send alerts whenever an error is found, so these can be addressed asap.

It's safe to say that Spidermon is essential for anyone working with web data scraping.

Combine automated data QA with manual processes

During this stage, datasets are analyzed to identify any potential sources of data corruption. If any issues are found, these are then manually inspected by the QA engineer.

So, always choose a team of experienced and dedicated QA engineers, to develop and execute manually-executed automated QA. This helps ensure that data is clean and accurate, and users are able to web scrape without dealing with any errors.

It's important that the quality assurance process is able to keep up to speed with the demands of a project. One of the most important parts of an automated data QA process are all of the manually executed data tests.

Implementing a series of data tests help identify potential issues and assign someone to solve them before they cause any major issues.

Overall, using a manually-executed process for automated data QA helps ensure that the system is functioning correctly and that users are able to use it flawlessly.

These tests repeatedly check data for consistency, providing an extra layer of protection against errors.

Catch bugs by using your eyes

The final step is to investigate any issues flagged by the automated data QA process and manually spot check sample sets of data comparing them against the scraped pages. This is done to validate that the automated QA steps haven’t missed any data issues and you receive everything that is expected from the extraction.

Visual QA cannot be fully automated, but there are tools that help do it more efficiently.

A type of quality assurance step here, is visual spotting of data inconsistency (literally).

This means to display a large sample of data that should be consistent (for instance product dimensions) and make use of the best possible tool for spotting oddities - your eyeballs.

These tests help catch any potential bugs before they cause problems for users.

Another example is, let's say there is a problem with the way the system handles dates, a QA team can process a manual test to catch the issue.

Once diligently deploying the steps above your dataset will be ready for efficient delivery of an automated data QA process.

Conclusion

In summary, the first and most important aspect is to ensure efficient deployment of your automated data QA process by applying rule-based Scrapy constructs, known as Pipelines.

Incorporating these help ensure you work with a high quality dataset.

Spidermon then identifies any errors that occur during the process to help fix them. If any issues are found, these are then inspected by the QA engineer.

The last thing you want is to be in the middle of a project, with data flowing in from various sources, and to deal with unexpected errors.