Manage and automate your spiders at scale
Schedule spiders with periodic jobs, follow every job's logs, requests and items as it runs, and control it all from the dashboard or the HTTP API
Scrapy Cloud
Run your spidersin the cloud
Deploy, schedule and monitor your spiders from one place. Scrapy Cloud handles the servers, so you focus on the data, not the infrastructure.
Create an account
pip install --upgrade shub && shub login && shub deployYou'll need your Scrapy Cloud API key and project ID
Capabilities
Everything your spiders need
Deploy, run, scale and check your Scrapy spiders in one place.
Focus on the output
Every item your spiders scrape is stored and browsable, ready to download as CSV, JSON, JSON Lines or XML, or to send to Amazon S3 and Google Cloud Storage with feed exports
On-demand scaling
Each unit adds 1 GB of memory, 2.5 GB of disk and another job running in parallel. Give a big crawl up to 6 units, then scale back when it's done
Your job status at a glance
Catch bad data before it reaches you. Monitor every job's item counts, field coverage and errors, and get alerted the moment something drifts
How it works
Local to the cloud in three steps
Create a free account
Sign up and create a Scrapy Cloud project. Your project ID is the number in its dashboard URL.
Deploy with shub
Install the shub command-line tool, log in, and deploy the Scrapy project you already have.
Log in with your Scrapy Cloud API key, not a Zyte API key.
Run and export
Hit Run in the dashboard, watch the job's logs and items as it goes, then download the data as CSV, JSON, JSON Lines or XML.
Pricing
Start free, scale as you need it
- Team members: Unlimited
- Projects: Unlimited
- Requests: Unlimited
- Crawl time: 1 hour
- Concurrent crawl: 1
- Unit resources: Half unit
- Data retention: 7 days
- Intelligent crawl scheduling: No
- Custom Docker images: No
- Support: Docs and Chat
Pro
$9per month,
per unit
Ideal for developers and companies who want a hassle free way to scrape the web at scale.
- Team members: Unlimited
- Projects: Unlimited
- Requests: Unlimited
- Crawl time: Unlimited
- Concurrent crawl: 1 per unit, up to unlimited
- Unit resources: 1 GB memory, 2.5 GB disk
- Data retention: 120 days
- Intelligent crawl scheduling: Yes
- Custom Docker images: Yes
- Support: Personalised
FAQ
Scrapy Cloud FAQ
Yes. The free plan gives you a low-resource unit with half the resources of a paid unit. Jobs can run for up to 1 hour and job data is kept for up to 7 days. Paid units cost $9 per unit per month and remove the runtime limit, keep job data for up to 120 days, and add job scheduling and custom Docker deployment. Students get one paid unit free.
A unit is the resource a job runs on: 1 computing unit, 1 GB of memory and 2.5 GB of disk. A job can use from 1 to 6 units, and resources scale with them, so a 2-unit job gets twice the memory and disk. You can run as many jobs in parallel as you have units; any others wait in a queue until units free up.
Install the shub command-line tool with pip install --upgrade shub, run shub login and paste your Scrapy Cloud API key (from your Zyte dashboard settings, not a Zyte API key), then run shub deploy with your project ID, which is the number in your project's dashboard URL. You can also deploy straight from a GitHub repository.
Yes. Scrapy Cloud is designed for Scrapy projects but supports other technologies. You can run Python scripts as jobs alongside your spiders, and scripts or crawlers in other languages can be deployed as custom Docker images that write items and logs through the Scrapy Cloud Write Entrypoint. Custom Docker deployment needs a paid unit.
Open a job in the dashboard and go to its Items tab to browse what it scraped, then export it as CSV, JSON, JSON Lines or XML. You can also fetch items programmatically through the Items API.