We’ve made a change. Scrapinghub is now Zyte! 

What is the difference between web scraping and web crawling?

time to read
By the one and only

People talk sometimes interchangeably about these two. But, actually, there's a difference. Want to know what is the difference between web scraping and web crawling? You're in the right place.

The short answer

The short answer is that web scraping is about extracting the data from one or more websites. While crawling is about finding or discovering URLs or links on the web.

Usually, in web data extraction projects, you need to combine crawling and scraping. So you first crawl - or discover - the URLs, download the HTML files, and then scrape the data from those files. This means you extract data and do something with it, like storing it in a database or further processing it.

Different purposes

Going deeper, there's a big difference in the purpose of these two things and how they work.

In web scraping, it's all about the data. The data fields you want to extract from specific websites. And it's a big difference because with scraping you usually know the target websites, you may not know the specific page URLs, but you know the domains at least.

With crawling, you probably don't know the specific URLs and you probably don't know the domains either. And this is the reason you crawl: you want to find the URLs. So that you can do something with them later. For example, search engines crawl the web so they can index pages and display them in the search results.

But another crawling example would be when you have one website that you want to extract data from - in this case you know the domain - but you don't have the page URLs of that specific website. So you don't know what pages to scrape. So first you create a crawler that will output all the page URLs that you care about - it can be pages in a specific category on the site or in specific parts of the website. Or maybe the URL needs to contain some kind of word for example and you collect all those URLs - and then you create a scraper that extracts predefined data fields from those pages.

Different outputs

So with web crawling the output is a lot more simple because it's just a list of URLs — I mean you can have other fields as well but the main elements are the URLs.

And with web scraping, you usually have a lot more fields 5-10-20 or more data fields. The URL can be one, but when you scrape, you extract the data not necessarily for the URL but for other data fields that are displayed on the website which can be - depends on the business use case - product name or product price, or some text or other information from any type of website.

Learn more about web scraping

Here at Zyte (formerly Scrapinghub), we have been in the web scraping industry for 12 years. We have helped extract web data for more than 1,000 clients ranging from Government Agencies and Fortune 100 companies to early-stage startups and individuals. During this time we gained a tremendous amount of experience and expertise in web data extraction. 

Here are some of our best resources if you want to deepen your web scraping knowledge:

Written by Colm Kenny