The Scrapy tutorial part II: How to create a spider in Scrapy

Developed by Pablo Hoffman and Shane Evans, Scrapy is an open-source python framework built specifically for web data extraction. With Scrapy spiders, you are able to download HTML, parse and process the data and save it in either CSV, JSON, or XML file formats.

In our first video, we showed you how to extract data from a webpage using Scrapy Shell. Now it's time to automate data extraction by building a Scrapy spider to visit the same page to fetch the current code from there.

After watching this, you will know:

• The anatomy of a Scrapy spider
• How to run a spider

If you haven't yet, we recommend watching part I of our tutorials first.

If you like what you saw, we can recommend you to also check out the links below:

• The Scrapy tutorial course part III: How to scrape multiple terms per page
• The Scrapy tutorial course part IV: How to follow pagination links
• Scrapy tutorial
• Scrapy CLI tool commands