Solution Overview
The Web Scraping Ecosystem for Professional Developers
Overcome any web scraping challenge quickly and efficiently with the Zyte coherent web scraping eco-system.
Write your spiders code with a scraping framework
We recommend you start with Scrapy an open-source web scraping framework for Python, created and maintained by Zyte. Check our learn Scrapy tutorials or join the Extract Community on discord to connect with web scraping experts.
from scrapy import Field, Item, Spider
class MyItem(Item):
price = Field()
class Scraper(Spider):
name = "scraper"
start_urls = ["https://the-best-store.com/"]
def parse(self, response, **kwargs):
item = MyItem()
item["price"] = response.css(".price_color::text").get()
return itemDeploy to Scrapy Cloud (optional)
Host, monitor and QA your Scrapy spiders in Scrapy Cloud the perfect solution to scale web scraping projects quickly and reliably with Zero vendor lock-in. Deploy code to Scrapy Cloud via your command line or directly with GitHub or access AI-Powered Smart Spider templates.
Handling bans and blocks
Configure Scrapy requests to use Zyte API to automatically extract data from websites of all complexity levels, using only the leanest tech to handle bans. Plus render Javascript, automate browser actions and take screenshots. Available as either a REST API or HTTP API.
def start_requests(self):
yield scrapy.Request(
url="https://quotes.toscrape.com/",
meta={
"zyte_api": {
"geolocation": "CA",
"browserHtml": True
}
}
)Turn web pages into JSON with AI Extraction
Switch on automatic extraction for articles, product pages or job listings and our patented ML will structure it for you. Forget about writing (and fixing) parsing code for your website.
Massively reduce time to build spiders
Minimize maintenance overhead per site
Can be extended and overridden
from scrapy import Request, Spider
class MySpider(Spider):
my_spider = "toscrape_com"
def start_requests(self):
yield Request(
"https://books.toscrape.com/",
meta={
"zyte_api_automap": {
"httpResponseBody": True,
"productList": True,
},
},
)
def parse(self, response):
http_response_body: bytes = response.body
productList = response.raw_api_response["productList"]Go 'All-in' on AI for product data
Our complete solution to extracting e-commerce product data with AI
Radically scale your web data with our end-to-end solution to web scraping products using AI.
Write e-commerce spiders 3x faster,
Generate 50% less maintenance overhead
Customise AI Extraction with Scrapy
Ready to scale up?
When you want to level up and tackle advanced use cases or scale quickly, technology alone often isn’t enough. With our Enterprise plans, we combine our leading technology with our industry expertise to give you:
Developer-to-developer training and support
Enhanced SLAs
Enterprise pricing and volume discounts
Outsource to Zyte
For organizations that want to outsource some (or all) of their web data collection, Zyte offers web scraping services backed by 13+ years of experience plus the largest team of 100+ web scraping experts. Let our experts understand your needs, and build the best solution to get the data for you.
