Swiss Army Knife Docker Container for Web Scraping

So relatively I'm new to the world of web scraping. But there's one thing that I have realized early on is that web scraping is more about researching your target than actually writing the code. Every website is different. Every data point is different. And before you diving into the project and doing the scraping, you actually need to understand what is the target website that you want to scrape. And over the last few weeks as I have been learning more about it, I found myself setting up the Python virtual environments, installing scrapey, beautiful soup and different sort of packages and framework to actually research my target. And uh there has been some mismatch in the versions. There has been some cleanup issues. So I decided to let's make a Swiss Army knife docker container for web scraping. Something that I can summon something that has all the packages required to get the web scraping research done. And once I'm confident after the research on my target, I can simply kill the Docker container. No need to worry about clean up. No need to worry about tear down. And whenever I want to do some uh research on the scraping on any website the next time, I can simply summon the container again. Making the job very much easier. So let me quickly show you how I put together this docker container. What are the different packages which is available in this container and how you can use it and also some quick examples on actually using and doing some research on the actual website. So let's go. All right. So this is the project. The link to the GitHub repo will be available in the video description. So you can actually clone it. You can actually uh use it and also contribute to it if you want to add any other package. Uh this is sort of a master container which has everything. So over here you can see I have installed scrapey. I have uh made site API available as a middleware. Then there is a request package for Python, beautiful soup, lxml. Uh there is pandas, jupyter notebooks. I think it's currently not working. I have to see that one out. But there is a curl utility, jq and then there is also nano and vim text editor just in case you need that. And uh I'll show you how you uh you can run it. I have also made it available on dockerup. So you can quickly fetch it and start using it. But let's also take a quick look at the docker file over here. So simply this is a base image the python 3.12 bullseye and these are all the packages that I have installed nanoim curl you can see the entire list and all the dependencies and then to make the zid API available I've also added this small particular script which will add the zite API uh in the settings py of the scraping uh of the scrapy project and yeah everything is from there so how you can use it let me show you in the terminal so first of all let's see you have docker installed that is a prerequest site yes we have docker installed now let's do docker pull And this is the image name my username I am pawa scraping Swiss army knife and the latest. So this will fetch the latest image from the docker hub. Uh you can also build it using this repo. So you can actually clone this repo and build it locally using the docker build command. Uh that also work and you can also remove any package that you won't need or add any package that you might need. But I'm actually very curious to know what packages you could add as well. So also feel free to share issues and pull requests and I'll take a look. Uh now docker container is uh installed. Let's take a quick look. So there should be the docker image. Yes, we have that one. Now let's run it. So what we can do is we can actually run it in interactive mode. And uh this is the container that we have. This is the Swiss Army knife web scripping container. Let's quickly verify few things over here. So let's see the version of Python. So Python is 3.12.1. So that's there. Let's also check the scrapey version. So that is also there. Scrapy version is also there. Uh let's uh do a quick uh you know sort of a research on a website. Let me scrape this example.com. So what I can do is I can type over here scrapey shell and then I can uh add the domain. So you can see that it's working. Uh we can also check it over here. So let's see let's do response dot status. You can see it's 200. So it's actually fetched. So let's quickly fetch the paragraph from this uh website using scrapey. So we can do response dot CSS and we can do over here paragraph. Uh let's do the uh text and let's call the get all function. So you can see the paragraph has been fed and the scrapy is working well. So scrapy works. Now let's see what other packages we have inside Python. So we can do a small uh request command. Let's do on or let's do on HTTP bin request dot get https http http httpbin/get. Let's do that and let's see the status code. Uh 200. Okay. Uh so yeah uh we can see that request is working. What else we have? We have beautiful soup. So from BS4 we can import beautiful soup. Uh we also have pandas. So yeah all the different libraries are available for you to do all sort of web scraping research work on any domain that you feel. And uh lastly I also want to show you that you can also run it uh with the uh zite API. So what you have to do you have to simply run this command where you also send a zite API key as environment variable. So the command is simple docker run in interactive mode and hyphen e is for the environment variable that will be inside the docker container and this is the variable that you need to store zite api key all in uppercase and test api is just for the placeholder over here and you need to replace it with your actual API key that you can find on zite but once you are in we can see that it now also has the uh let's say uh zite API key so it should be here so yes we have the zite API key in the environment and you can use it alongside scrapy so That is the Swiss army knife of web scraping a docker container. Uh again you can al also fetch it and build it locally. You just have to use the docker build command. You can remove the packages you do not want. You can add the package that you might need and do all sort of stuff. Uh again this is not something that you will be want to use in production. This is something that I will be using for all my web scripping research work. But yeah happy to take pull request, happy to take issues and you know keep this going. Also let me know in the comment section if you want to see any other package to be added on this particular container. But yeah, you can also remove the uh image like this. So you can fetch it uh latest one or you can build locally. But before that do remove it. But yeah, that's all I wanted to showcase you in this video. Let me know what you think about this one. Any other package that you want me to add, do comment it down below and subscribe to the channel if you haven't already. And do join our Discord channel for more interesting web scraping conversations and projects. I'll see you in the next one.