The reputation of web scraping hasn’t always been the best. Unsavory actors have cast a shadow over the reputable parts of the web scraping industry at large, and it has to stop.

I frequently talk to attorneys at big companies who believe that web scraping is illegal despite the fact that the largest businesses in the world are using public web data to make crucial business decisions.

Nebulous, vague and misunderstood? Yes. 

Illegal? Absolutely not.

Tangible guidelines and best practices don't exist in an official capacity, and a lot of companies are doing things they shouldn't. The industry is fraught with opaque laws that are coupled with strong data protection from governments. It’s not always clear which is the best way to navigate running an ethical data practice.

Zyte’s commitment to ethical web scraping has been the foundation of our business since the beginning, so I jumped at the chance to help bring maturity to our industry by co-founding the Ethical Web Data Collection Initiative (EWDCI).

The EWDCI and i2Coalition launched a certification program, and it's a massive step forward in maturity for our industry. An EWDCI certified designation shows the world that the company adheres to the agreed-upon principles and the highest degree of ethics when collecting web data. I’m pleased that Zyte is part of the inaugural group of companies to achieve the certification.

What’s the EWDCI

The EWDCI is a consortium of web data extraction companies that are focused on:

• strengthening public trust,

• promoting ethical guidelines, and

• helping businesses make informed choices.

The consortium aims to bring ethics to the web scraping process to promote a key component of a free, fair, and open internet — trust. I’ve been a hands-on part of the EWDCI since the beginning, and one of the first things we collaborated on was creating a core set of web scraping principles that cover aspects of: 

• legality, 

• ethics, 

• ecosystem engagement, and

• social responsibility. 

We then launched the certification program to bring more accountability and transparency to the industry, as well as build consumer confidence around public web data collection. Companies that receive the EWDCI certification have agreed to follow the four principles and maintain the highest level of ethics.

Why the EWDCI Is Needed

Our industry is full of mistrust and misconceptions about web scraping. I felt that Zyte, with our heavy experience in ethical web data extraction, needed to work on changing this narrative. I jumped at the chance to help shape these narratives with other big players in the web extraction community like Oxylabs, Rayobyte, and Smartproxy. I want to show that there are a lot of companies out there who care about:

• the law,

• Ethics, and

• your data and what companies are doing with it. 

Zyte and these companies are building principles to ensure that there becomes a level of public trust in collected public data and in web data extraction companies. I genuinely believe not only is it crucial for Zyte, but also for the whole ecosystem.

There are a lot of web data extraction companies out there who are doing a lot of things they shouldn't. I think if we can all get on the same page and collectively say, “Hey, This is what we think the right way to do this is, even though the laws are confusing, and even though there's no perfect, right answer?” 

At Zyte, we do everything we can to be ethical in our practices, and I knew that our expertise needed to be present at the table that’s going to shape the future of our industry. There aren’t many opportunities to push an immature industry towards ethical and trustworthy practices, and I wanted my team and Zyte to be a part of that.

Why Businesses Need Educating

With the EWDCI and the certification, we can create more trust with businesses. My team hasn't had a case where we weren’t able to convince the lawyer or senior executive that scraping in an ethical manner is indeed legal, but if we had established standards, it would be much easier. We would have something to point to say, “Hey, these are actually the standards, we follow them, and we're actually co-authors of them.” I think that's a very powerful message when you're trying to sell business to business and get people to get comfortable with the idea and potential of web data extraction. 

Why Certify

The EWDCI certification is a welcome signal to customers, partners, and the public that you operate a web data extraction business that operates at the highest degree of ethics. This certification makes it easier for us to reassure businesses that they can take advantage of web data without the negative perception and dodgy scraping practices. If you want to learn more about and how to apply for the EWDCI certification be sure to reach out to the i2Coalition Membership Director.