Scrapy Cloud

Introducing Web Scraping Copilot - A rocket boost for data extractors

Meet Web Scraping Copilot, a free VS Code extension that uses AI to accelerate Scrapy projects. Generate code, manage spiders, and deploy to Scrapy Cloud faster than ever, keeping developers in control.

Scrapy Cloud secrets: Hub Crawl Frontier and how to use it

Imagine a long crawling process, like extracting data from a website for a whole month. We can start it and leave it running until we get the results.

Scrapy Cloud Secrets: Hub Crawl Frontier And How To Use It

Imagine a long crawling process, like extracting data from a website for a whole month. We can start it and leave it running until we get the results.

Deploy Your Scrapy Spiders From GitHub | Scrapy Cloud

Up until now, your deployment process using Scrapy Cloud has probably been something like this: code and test your spiders locally, commit and push your changes to a GitHub repository, and finally deploy them to Scrapy Cloud using shub deploy.

How To Run Python Scripts In Scrapy Cloud

You can deploy, run, and maintain control over your Scrapy spiders in Scrapy Cloud, our production environment.

Introducing Scrapy Cloud with Python 3 support

It’s the end of an era. Python 2 is on its way out with only a few security and bug fixes forthcoming from now until its official retirement in 2020.

Introducing Scrapy Cloud 2.0

Frontera: The Brain Behind The Crawls

At Zyte we're always building and running large crawls–last year we had 11 billion requests made on Scrapy Cloud alone.

Scrape Data Visually With Portia And Scrapy Cloud

In case you aren’t familiar with Portia, it’s an open-source tool we developed for visually scraping websites. Portia allows you to make templates of pages you want to scrape and uses those templates to create a spider to scrape similar pages.

New Changes to Our Scrapy Cloud Platform: Enhanced Performance and Features

We are proud to announce some exciting changes we've introduced this week. These changes bring a much more pleasant user experience, and several new features including the addition of Portia to our platform!