How to create a Docker container with Scrapy and PostgreSQL

All right. So, continuing my experiments with Docker and web scraping. Today I want to showcase you a new project in which using Docker Compose we will be scraping websites and storing the data in Postgress SQL database. Uh each service the spider will be running as a separate docker container. The database that is the Postgress SQL will be running as a separate docker container and they will be talking to each other using the scrapey pipelines and it's a very fun clean beginner project that showcase you the power of the scrapey framework. how you can handle pagionation in the scrapey. How you can uh uh scrape different websites and store the data that you need into a database and this also is very scalable project because the scrapy spider runs as an independent service as a script. So only when you want to run it, you can uh invoke it and the database can run continuously. It can also store data in your host computer using docker volumes. So let's dive right in. First of all, let me showcase you how the project works and then I'll also give you a brief walk through the code. So to make things a little easier, I have created a make file. Uh so I don't have to write the entire docker commands again and again. So first of all, I want to invoke the DB. That means I want to make sure that my Postgress docker container is running and the make DB command ensures that. Uh next I want to scrape a website and the good thing is you can give the URL uh in the command and argument. So let's scrape something which is from books to scrape a website. And what I want to scrape I want to showcase you that how it handle pagenation. So let's scrape a category that has a different page. Philosophy mystery. Yes, mystery has at least I think two different pages. So that's going to be good. So I'm just copying this URL and I will be pasting this URL over here. And I can also give how many pages I want to scrape. But uh I will not showcase you for this one. Let's just do the scrape. And this will create another docker container which has the scrapey spider. So it will go out and it will first make sure that the database is ready. And uh each scraped item the title of the book and the uh price of the book is stored in the in the database. And uh as we can see that I think it's pretty much successful. You can see 200 200 you can see the prices the title of these different books and uh this particular category has total 32 results. So 20 on this one and the remaining one on the next one. Uh so we should see 32 books in our database. Uh let's check that. So to check the database I have made this command p SQL which which is for postress SQL. So now I can check if all the books are there. So select star from books. Books is the name of the DB and you can see there are indeed 32 rows. Sharp object is the name of the book which is for 47.82. Let's just confirm over here. Uh so if we go to the first page we can see the sharp object and indeed everything is looking good and we got all the 32 rows over here uh inside our DB and you can do any sort of SQL commands over here because this is just a Postgress DB. Uh so yeah that's uh how it works and if you look at the docker ps you will only see that only the post crest is running uh you will not see the spider is running because the spider will only run when you want it. So if I have to do uh let's say another URL so I can simply do make scrape URL equal to and now we can give it something different category let's give it a music which has another 13 results and it should add another 13 uh entries to our DB. So it's now going to be fetching the spider is working and the spider after it's completing the fetch the scripping it's you know it's finished the container is killed. So again if I now do make psql and select star from books oops select star from books. So uh now you can see that it also has a new entry over here which reflects the page that we have just scraped. So yeah, this is how you can make your scraping projects and because it has different Docker containers for the database for the spider, it's really easy to manage and scale. Now let me quickly showcase you how it's working. I also showcase you the docker file and the docker compos file which makes it all possible. Zooming in over a bit and uh let's first talk about the docker compos file because that's the most important file. So as I mentioned there are two services over here. One is the postcress. So I have kept everything same. I haven't changed the username and the password. uh even the port is the default uh port and the docker volumes is given because even after the container is scaled you will still have the data saved using the docker volumes in your host computer. So this is how you can set up the postgress and the good part is if you want to use a different database for example MongoDB then you can change it and have the image fetched from the docker hub. Similarly this is the scrapey container uh this one has your spider and this is being built locally. I'll show you the docker file what's inside because it's docker composer and networking is handled inside using the container name itself. So we don't have to worry about it. One thing that you can see over here is restart is equal to no. That means we only want to run the scraper when we manually invoke it. Otherwise it stays dormant. The container is not run. Uh now let me show you the docker file. So docker file is to build the uh the scraper. So just a simple small docker file where we have everything we are installing scrapey and then we have a small script which is running the spider and the script what it does it's taking the URL from the command and argument and then just invoking scrapey crawl and books is the name of the spider where all the crawling logic is. So let's also take a look at our spider. If you go to the book scraper spider and you see books.py so that is the scrapey spider that we have created for this one. uh simply we are taking the URL from the command and arguments and that we are passing over here and over here we are handling the pagenation as you can see and what we are doing is we are actually just fetching the title and the price that's what we want to save but if you want to save any other data then you can feel free to add more lines over here and uh one thing that you need to uh take care over here is that you need to yield the title and the price so what is happening over here is scrapey after every successful scraped item it's invoking the next in the pipeline which is the Postgress SQL database and I'll show you how that works as well. So this is a scripping logic. This is the spider and the important thing over here is if I go to the pipelines py over here you can see we have created a postgress pipeline and what this does it's actually using the postgress host name the database username and the password which we have shared in the docker compost file and uh using this logic whenever there is one successful scraped item it's creating a new entry to to the database. So that's how it works and in settings.py we have to enable it. So we have enabled the pipeline book scraper pipeline postgress pipeline and we have given this dictionary 300 priority. So that's how it all works together seamlessly. Scrape is a very powerful framework and uh if you know what you are doing then it's definitely makes it very easy to scrape web handle pagination handle pipeline. Uh you can also do different things with the pipeline. You can maybe add another database you can change some data if you want. For example increase price everything by 10%. You can all do in the pipelines thing. So yeah, that's a sample project that I wanted to showcase you how you can leverage docker compost to create the scrapey spiders and store the data in your preferred database. Uh the entire project is available on GitHub. I will be adding the link in the description so you can check it out and modify according to your needs and also read the blog if you need more information on how everything is working. But let me know in the comment section what other projects you want me to make and showcase you. The next thing I'm planning to add in this project is to enable Zite API so it can also handle JavaScript well and also handle the antibbans. But yeah, do join our discord community if you have any questions and people there would definitely love to help you in your web scripping journey and I'll see you in the next