We’ve made a change. Scrapinghub is now Zyte! 

Dirbot: A new example Scrapy project

time to read
< 1
Mins
By the one and only
April 28, 2011

Scrapy users have complained in the past about the lack of a pre-built example project that contains, for example, the dmoz spider described in the tutorial.

Complain no more!. We're happy to let you know that there is a now functional Scrapy project available on Github which contains the old Google Directory spider and the Dmoz spider described in the tutorial.

The project is called "dirbot", and it's available at https://github.com/scrapy/dirbot

The documentation of Scrapy 0.13 (which will become the next stable release, Scrapy 0.14) has been updated to point to this new example project.

Written by Kevin McKinless
Web scraping specialist with over 10 years experience. An expert in Python and Rocket League. Join me on social media and we can talk all things Data Extraction.
Sign up to the blog