Here are some questions to ask your team and partners that can help you with your web scraping scaling journey.

Questions to ask the stakeholders

• What business problem are we trying to solve with web data?

• How does this align with our key business objectives?

• Are you aware of the potential challenges and risks associated with web scraping at scale?

• Will this effort consider initial capital investment and the required ongoing funding?

• What’s the budget?

• What is the timeline of this effort?

• What is the priority of this effort compared to other initiatives?

• Are we currently operating in a legal and ethical manner?

• How well does our legal team understand the complexities of web scraping? 

• Do we have resources who are experienced with managing large web scraping operations?

• What are your expectations for return on investment for this effort?

• Where do you see web scraping at scale in the business evolving in the next 5 years?

Questions to ask your QA and development teams

• What tooling is part of our web scraping tech stack?

• How scalable is our tooling? If it’s not scalable, what would we need to do to change that?

• Do we have resources who are experienced with managing and maintaining large web scraping operations?

• Do we have enough resources to scale?

• What challenges have we encountered so far and how did we overcome them?

• Do we have the infrastructure required to scale? If not, what is needed?

• How are we currently managing website bans and how would these strategies change at scale?

• How is the team currently managing web scraping projects?

• What’s our current end-to-end workflow from development, testing, deployment, and maintenance?

• How will scaling affect ongoing projects and priorities?

• Do you trust data from targeted websites? 

• How do you find reliable and quality web data?

• What happens when you find conflicting data gathered from multiple sources?

Questions to ask a third-party

• What industries have you worked with?

• What data have you been asked to extract?

• What is your web scraping tech stack?

• How do you handle website changes and anti-bot technology?

• How do you ensure the quality of the data at large volumes?

• How do you ensure legal compliance within your scraping projects?

• What ethical guidelines do you follow?

• Are you certified by the EWDCI?

• What’s your project management approach?

• How do you communicate with clients?

• What is your pricing structure and additional costs?

• What is your security strategy to protect the data collected?

• How do you handle security incidents?

• What are your support strategies and SLAs?

Scaling a web scraping operation has a lot of moving parts. You can talk to our web scraping specialists to help you build a stronger business case to get the best results from your scaling efforts. And when scaling in-house, your development team should sign up for a free trial to Zyte API. They can get hands-on experience with an end-to-end tool for crawling, unblocking and extracting data that can help your business scale.