Compliant Web Scraping Checklist
DISCLAIMER: This post is for informational purposes only. The content is not legal advice and does not create an attorney-client relationship.
Is your web scraping project legal? Here’s a checklist to help you answer that question.
Because there are no specific web scraping regulations, there’s a labyrinth of laws that one must navigate before embarking on a web scraping project. Below is a checklist to help you get started on your web scraping journey in the most legally compliant and ethical manner. If you answer “Yes” to any of the Items in the checklist below, you’ll want to follow the steps outlined in the Details column and consult with your own lawyer to ensure compliance. If you answer “No”, then your project is likely much lower risk.
We hope this helps you navigate the legality of your project and if you aren’t sure how to answer an Item on the list or you answer “Yes” and don’t know what to do next, Zyte has a team of legal and compliance scraping experts who can help guide you on your web scraping compliance journey. Just reach out at legal@zyte.com.
| Yes or No | Item | Details on Next Steps |
|---|---|---|
| - | Non-Public Data | If the data is not publicly available on the internet, meaning a regular member of the public cannot simply navigate to the page without some sort of authentication, such as a login or pay wall, then you must carefully consider whether you should be accessing this data. In most cases you should not access it without permission or a full review of the website terms to ensure compliance with those terms. |
| - | Explicit Agreement to Terms | If you explicitly agree to the website’s terms of services or other policies in any way, meaning clicking ok or agreeing to the policies, checking a box, or any other explicit action to agree to the terms, then you must abide by the policies that you have explicitly agreed to. Read the relevant policies in their entirety in order to determine what action you may take on the website and what you may or may not do with the data. |
| - | Mobile App Download | When downloading a mobile app, check which contractual agreements (such as privacy policies) you are agreeing to when downloading. Read the policies you are agreeing to in their entirety in order to determine what action you may take on the mobile app and what you may or may not do with the data. |
| - | Copyrighted Data | Determine what fields of data you are scraping and if any of the data is protected by copyright. If it is copyrighted data, determine if your use constitutes copyright infringement, and if so, whether it falls under an exception to copyright law (such as fair use). Descope the copyrighted data if no exception applies. |
| - | Personal Data | Determine what fields you are scraping and if any of the data is considered personal data. If the data includes personal data, ensure your collection and use of the data is compliant with the relevant data protection laws. Descope or anonymize the personal data if you are not able to satisfy the requirements under the applicable data protection laws. In the EU you will need a lawful basis to process personal data and in the US if it is public personal data it likely falls under the exceptions within the various state laws. Note that certain types of personal data, such as health information or sensitive personal data, will be held to a higher standard under many data protection laws. |
| - | Improperly Sourced IP Addresses | If you are sourcing IPs directly or using a scraping provider that uses IPs to obtain the data or within their software, ensure that the IPs are obtained legally and ethically. Most importantly, if residential IPs are being used, ensure that they are collected in compliance with applicable data protection laws and you are using a reputable provider that do not allow improper use of the residential IPs. |
| - | External Use of Data | Most scraped data should only be used for internal business analysis. If you plan to use the data externally, ensure that your use is compliant. This will be a very fact specific analysis and will need to be taken on a case by case basis. |