Scrapy Cloud
Scrapy Cloud
Focus on crawlers, not servers
Manage spiders, not servers.
Any number of spiders. Any team size.
Manage and automate spiders at scale.
Scrapy Cloud is a battle-tested cloud platform for running web crawlers (aka. spiders). Your spiders run in the cloud and scale on demand, from thousands to billions of pages. Think of it as a Heroku for web crawling.
Over 2,000 companies trust Scrapy Cloud to run their spiders
Make managing spiders a breeze
Run, monitor, and control your crawlers with Scrapy Cloud's easy-to-use web interface.
On-demand scaling
Increase the scale and firepower of your scraping operation with only a few clicks.
Easy integration
Seamlessly integrate Crawlera, Splash, Spidermon, etc. to your web scraping stack to take the hassle out of scraping the web at scale.
Full suite QA tools
Built in spider monitoring, logging and data QA tools. Along with easy integration of Spidermon, our open source spider monitoring framework.
Zero vendor lock-in
Develop your code using Scrapy, the most popular open-source web scraping framework, and retain the freedom to migrate it to any hosting solution.
Start scraping the web in minutes
Deploy code to Scrapy Cloud via your command line or directly with GitHub.
Terminal
pip install shub shub login API key: (api_key)
shub deploy (project_ID)
Everything You Need Straight Out of the Box
Scrapy cloud has been specifically design for web scraping at scale
Real-time dashboard
Intelligent scheduling
Built-in
monitoring
Customisable containers
Crawlera integration
Splash
integration
Elastic Pricing
Only pay for as much capacity as you need. Simply purchase capacity units, 1 concurrent crawl, when you need to scale up.
Starter
Ideal for small projects or if you simply want to give Scrapy Cloud a try.
Unlimited team members
Unlimited projects
Unlimited requests
1 hour crawl time
1 concurrent crawl
7 day data retention
Professional
Ideal for developers and companies who want a hassle free way to scrape the web at scale.
Unlimited team members
Unlimited projects
Unlimited requests
Unlimited crawl runtime
Unlimited concurrent crawls
Intelligent crawl scheduling
120 day data retention
Personalised suppport
1 Scrapy Unit = 1 GB of RAM and 1 concurrent crawl
With Scrapy Cloud, you control how you allocate your resources. Just update your settings to allocate 1 Scrapy Cloud unit to each job.
Compare plans to find the right solution for your needs or talk to one of our solution architects today.
Live chat
