Explore resources by topic or category

When crawling the web, there’s always a speed limit. A spider can't fetch faster than the host willing to send the pages.

As a python developer at Zyte (formerly Scrapinghub), I spend a lot of time in the Scrapy shell.

If you’ve been using Scrapy for any period of time, you know the capabilities a well-designed Scrapy spider can give you.

If you know anything about Zyte , you know that we are obsessed with data quality and data reliability.

Absolutely not! Website changes (sometimes very subtly), anti-bot countermeasures, and temporary problems often reduce the quality and reliability of our data.

This is a guest post from the folks over at Intoli, one of the awesome companies providing Scrapy commercial support and longtime Scrapy fans.

The first rule of web crawling is you do not harm the website. The second rule of web crawling is you do NOT harm the website. We’re supporters of the democratization of web data, but not at the expense of the website’s owners.

Welcome to This Month in Open Source at Zyte! In this regular column, we share all the latest updates on our open source projects including Scrapy, Splash, Portia, and Frontera.

We eat our own spider food since Scrapy is our go-to workhorse on a daily basis. However, there are certain situations where Scrapy can be overkill and that’s when we use Parsel.

Many governments worldwide have laws enforcing them to publish their expenses, contracts, decisions, and so forth, on the web.