Scale up your Scrapy projects with Smart Proxy Manager

In this tutorial, you will learn how to use smart proxy manager to scale up your already existing Scrapy project in order to make more requests and extract more web data.

Scrapy is a very popular web crawling framework and can make your life so much easier if you’re a web data extraction developer. Scrapy can handle many web scraping jobs including URL discovery, parsing, data cleaning, custom data pipelines, etc… But there’s one thing that Scrapy cannot do out of the box and it has become a must if you want to extract large amounts of data reliably: proxy management.

In order to scale up your Scrapy project, you need a proxy solution.

I will show you how to turn your already existing Scrapy spider and boost it with proxies!

Getting started

For this example, I’m going to use the “Scrapy version” of the product extractor spider that contains two functions:

1. A crawler to find product URLs
2. A scraper that will actually extract the data

Here’s the Scrapy spider code:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

class ProductSpider(CrawlSpider):

name = 'product'

start_urls = ['http://books.toscrape.com/catalogue/category/books/travel\_2/index.html'\]

rules = (

Rule(LinkExtractor(restrict_css='article.product_pod > h3 > a'), callback='populate_item'),

)

def populate_item(self, response):

item = ProductItem()

book_title = response.css('div.product_main > h1::text').get()

price_text = response.css('p.price_color::text').get()

stock_info = response.css('p.availability').get()

item = {

'title': book_title,

'price': self.clean_price(price_text),

'stock': self.clean_stock(stock_info)

}

yield item

def clean_price(self, price_text):

return Price.fromstring(price_text).amount_float

def clean_stock(self, stock_info):

return remove_tags(stock_info).strip()

class ProductSpider(CrawlSpider): name = 'product' start_urls = ['http://books.toscrape.com/catalogue/category/books/travel\_2/index.html'\] rules = ( Rule(LinkExtractor(restrict_css='article.product_pod > h3 > a'), callback='populate_item'), ) def populate_item(self, response): item = ProductItem() book_title = response.css('div.product_main > h1::text').get() price_text = response.css('p.price_color::text').get() stock_info = response.css('p.availability').get() item = { 'title': book_title, 'price': self.clean_price(price_text), 'stock': self.clean_stock(stock_info) } yield item def clean_price(self, price_text): return Price.fromstring(price_text).amount_float def clean_stock(self, stock_info): return remove_tags(stock_info).strip()

1class ProductSpider(CrawlSpider):
2	name = 'product'
3	start\_urls = \['http://books.toscrape.com/catalogue/category/books/travel\_2/index.html'\]
4	rules = (
5    		Rule(LinkExtractor(restrict\_css='article.product\_pod > h3 > a'), callback='populate\_item'),
6	)
7	def populate\_item(self, response):
8    	item = ProductItem()
9    	book\_title = response.css('div.product\_main > h1::text').get()
10    	price\_text = response.css('p.price\_color::text').get()
11    	stock\_info = response.css('p.availability').get()
12    	item = {
13        	'title': book\_title,
14        	'price': self.clean\_price(price\_text),
15        	'stock': self.clean\_stock(stock\_info)
16    	}
17    	yield item
18	def clean\_price(self, price\_text):
19    		return Price.fromstring(price\_text).amount\_float
20	def clean\_stock(self, stock\_info):
21    		return remove\_tags(stock\_info).strip()

So let’s assume this is the spider you currently have and it was working fine… delivering you precious data… for a while. Then you wanted to scale up and make more requests which ultimately led to blocks, low success rate, bad data quality, etc… 

The solution to overcome blocks and receive high-quality web data is proxies and how you use those proxies. Let’s see how you can integrate Smart Proxy Manager in this spider!

Scrapy + Smart Proxy Manager

The recommended way for integration is the official middleware. This is how you can install it:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

pip install scrapy-crawlera

pip install scrapy-crawlera

1pip install scrapy-crawlera

Then, add these settings in your Scrapy project file:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

DOWNLOADER_MIDDLEWARES = {'scrapy_crawlera.CrawleraMiddleware': 610}

CRAWLERA_ENABLED = True

CRAWLERA_APIKEY = ''

DOWNLOADER_MIDDLEWARES = {'scrapy_crawlera.CrawleraMiddleware': 610} CRAWLERA_ENABLED = True CRAWLERA_APIKEY = ''

1DOWNLOADER\_MIDDLEWARES = {'scrapy\_crawlera.CrawleraMiddleware': 610}
2CRAWLERA\_ENABLED = True
3CRAWLERA\_APIKEY = '<API key>'

Notice that in order to use Smart Proxy Manager, you need an API key. But don’t worry, we offer a 14-day free trial (max 10K requests during the trial) so you can get your API key fast and try it out to make sure it works for you.

Optionally, you can also set the proxy URL if you requested a custom instance of Smart Proxy Manager:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

CRAWLERA_URL = 'myinstance.zyte.com:8011'

Another way to set up Smart Proxy Manager is directly in the spider, like this:

class ProductSpider(CrawlSpider):

crawlera_enabled = True

crawlera_apikey = 'apikey'

CRAWLERA_URL = 'myinstance.zyte.com:8011' Another way to set up Smart Proxy Manager is directly in the spider, like this: class ProductSpider(CrawlSpider): crawlera_enabled = True crawlera_apikey = 'apikey'

1CRAWLERA\_URL = 'myinstance.zyte.com:8011'
2Another way to set up Smart Proxy Manager is directly in the spider, like this:
3class ProductSpider(CrawlSpider):
4	crawlera\_enabled = True
5	crawlera\_apikey = 'apikey'

Settings recommendations

To achieve higher crawl rates when using Smart Proxy Manager, we recommend disabling the Auto Throttle addon and increasing the maximum number of concurrent requests. You may also want to increase the download timeout. Here is a list of settings that achieve that purpose:

Plain text

Copy to clipboard

Open code in new window

EnlighterJS 3 Syntax Highlighter

CONCURRENT_REQUESTS = 32

CONCURRENT_REQUESTS_PER_DOMAIN = 32

AUTOTHROTTLE_ENABLED = False

DOWNLOAD_TIMEOUT = 600

CONCURRENT_REQUESTS = 32 CONCURRENT_REQUESTS_PER_DOMAIN = 32 AUTOTHROTTLE_ENABLED = False DOWNLOAD_TIMEOUT = 600

1CONCURRENT\_REQUESTS = 32
2CONCURRENT\_REQUESTS\_PER\_DOMAIN = 32
3AUTOTHROTTLE\_ENABLED = False
4DOWNLOAD\_TIMEOUT = 600

No more failed web scraping projects

Sign up for a Zyte Smart Proxy Manager account

This simple integration takes care of everything you need to scale up your Scrapy project with proxies.

If you are tired of blocks or managing different proxy providers, try Smart Proxy Manager .