Announcing Smart Scraping Beta for Zyte API

One of the highlights of Extract Summit 2023 was the beta launch of a revolutionary web data extractionsolution that redefines how developers and non-developers collaborate, leveraging AI to gather valuable information from the web.

There are two key takeaways from this announcement:

• Developers can now create spiders as reusable templates, with the solution generating a user-friendly UI.

• Zyte provides an editable e-commerce scraping template powered by AI, automating crawling, parsing, and ban handling for the user.

Using this solution and the included template, users can start extracting data from e-commerce websites within seconds, without the need for ongoing maintenance, thanks to AI-powered functionality.

To give it a try simply login and navigate to Scrapy Cloud, and follow the instructions to create a new project with Zyte's AI-Powered Spiders beta.

So what does the AI template do?

What are the major time consuming and maintenance heavy aspects of scraping?

• Handling Bans

• Creating Crawl Patterns

• Writing Parsing Code

• Fixing all of these fragile things when they break

With Smart Scraping for Zyte API you can access the infinite scale of AI with the control and quality of custom code

By using Zyte API as the foundation and Scrapy as the framework, the Smart Scraping templates and associated libraries automate many of the tedious, time-consuming tasks developers usually have to handle around bans, parsing and crawling. This solution takes care of the grunt work, so you don't have to.

But isn't AI expensive and low quality?

In the past, we might have agreed that AI wasn't economically viable and lacked the reliability needed for commercial-grade data collection. However, with the launch of Smart Scraping for Zyte API, we've made it possible for developers to use AI for web scraping when:

• You want to obtain data quickly.

• You want to keep collecting data without fixing code.

• You want data from many sites.

When the setup and maintenance costs of writing custom code outweigh the value of collecting data, AI becomes the key to success. We've worked on this for more than four years now - reducing costs and building models that can match or surpass human scraping. 

Battle tested by our services division

We know it works because we're our own biggest customer. We've found we can deliver data three times faster at the same quality for new websites, while generating radically fewer maintenance overheads.

What if I want to modify templates or create my own templates?

We've made it easy to modify and create your own custom spider templates using our Scrapy spider template as a base, or start entirely from scratch if you prefer.

• Expand the schema

• Send data to a third-party service

• Filter results and customize crawl patterns

• And much more.

How can you try Smart Scraping for yourself?

The great news is that an Open Beta will be available starting October 26, 2023, and you can use free trials to test the solution.

You will need two things to get started:

• A Scrapy Cloud account (which includes 1 free unit).

• A Zyte API account (which includes $5 free credit).

We believe this is the future of web data extraction, where technical and non-technical minds collaborate seamlessly with AI to harvest data from the digital world.

To get started simply signup for a free Scrapy Cloud account, and follow the instructions to create a new project with Zyte's AI-Powered Spiders beta.

PS. Currently, AI-powered templates are limited to e-commerce websites, but templates for news, media, and many other data archetypes will be coming soon.