The Scrapy tutorial course part I: How to get started with web scraping using Scrapy

Developed by Pablo Hoffman and Shane Evans, Scrapy is an open-source python framework built specifically for web data extraction. With Scrapy spiders, you are able to download HTML, parse and process the data, and save it in either CSV, JSON, or XML file formats.

Does Scrapy use proxies?

By default scrapy does not use proxies. However, as a web scraping framework it has the ability to easily plug in multiple proxy services or allow a user to easily write custom code for using a bespoke proxy using the scrapy’s default HTTP Proxy downloader middleware. Services like Zyte Smart Proxy Manager (formally crawlera) are built for easy use with scrapy and provide open source middlewares to easily incorporate their proxies into your scraping project.

This video covers the basics of web scraping using your web browser, Scrapy shell, and CSS selectors.

After watching this, you will know:

• How to identify the data via browser's "inspect element" tool
• How to build CSS selectors using Scrapy Shell

If you like what you saw, we can recommend you to also check out the links below:

• The Scrapy tutorial course part II: How to create your first Scrapy spider
• The Scrapy tutorial course part III: How to scrape multiple items per page
• Scrapy tutorial
• The 30 CSS selectors you must memorize
• An interactive tutorial on CSS selectors