Why Does No one use this KILLER Scrapy Addon?

If you were building an API or a web app, you wouldn't put loads of business logic and a load of database logic within the roots. You'd let them handle the request and the responses and that would be it. So why is it with Scrapey we insist on having the spider actually do all the passing and control the passing logic too? It doesn't make a lot of sense to me. So in this video I want to demo and show you scrapey poet which is scrapeyy's integration of web poet which allows us to have access to a page object and this gives us that opportunity to split out separation of concerns split out the passing logic from the spider. So I'm going to show you what I've got here and I'm going to explain as I go what it all does. So the first thing that we're actually looking at here is my spider. This is the products.py spider that I've created. takes in a URL and then you know we start and then we do our callbacks. But you'll notice right away that our pass function here that has this callback actually takes in an item. It takes in the item list page and it returns from that the product URLs and the response.follow and our pass item again takes in the actual product item and then returns that out. What this means is that our spider is now very very neat, very concise and it does very specific thing. It gets the data, it crawls the pages, and it sends it back to these functions that take in these items. But of course, we're just giving it an item. What are we actually doing with it? It's not going to know how to get that information. And that's where Scrapey perk comes in with the page objects. So, I'm going to look at the list item page first. And you'll notice here, I'll make this a little bit bigger, that we have this list page class. What this does is it's defined as a webpage object, which means it gives us access to say, hey, here's the product URLs that I want. This is the field that we're going to populate for this item. So, this is the same item that I created, and I'll show you that in just a second. It's going to return via this selector all of the links that it finds and then populate them into that item. So, if I look at the item, we have that here. Very straightforward. Data classes, very neat and tidy. keeps everything separate. It's all going to be here. All your items here. Now, this is the product page page object. So, this is exactly the same thing, but there's a lot more going on here. And that's because, you know, there's more fields to handle. So, we have a quick look at this. We have the same sort of thing. Web poet contains a page, our web page here, and returns the item. So, it knows what it's doing. We're also saying we're only going to handle the URLs in this item, this page object that match this. I also have this function here which is just to do some data cleaning. Uh I can put this in here as processes like so our processes class. Now these are the most important parts the fields here. So each field represents the field of the item and then we're going to say we're going to return this piece of information that comes from the web page via the HTML that we're passing out to actually get that data out and load it into the item itself. We don't have to do anything fancy. We it's all handled for us. We know now that we have this page object that matches our our item class that when the data comes back from scrapey spiders, it goes through here, it gets collected and then loaded to the item and yielded out from our spider. So what does this actually give us then? Well, first of all, it gives us much neater and much more maintainable code. If we think that perhaps you know maybe select some of the selectors change or you want to add a field in it's as simple as adding it into the item class and then adding in a field into your uh actual page object class. You can just keep adding them or changing them or you know changing the selectors as you need. And on the subject of selectors we can actually test against this class. Now we we can download the HTML via Piest, create some fixtures, and then run tests against the selectors that we have with the data that we downloaded. This means that we can stay on top of our spider maintenance much much easier. It also means that when you're working within a team, collaborating and creating spiders is much easier because it's all handled in the same way. There's going to be for every page type, there'll be a page object and an item that will be yielded through the spider. All these things are separated out easy to manage easy to maintain and it's very repeatable and anyone coming into your team can learn this and understand how it works and then utilize it and help right away. So to get this to work is very very straightforward. You just need to install the add-on called scrapey poet and in your settings you need to enable it here and then add in this command the scrapey poet discover which points to a folder in your project. This is my folder up here. It's called pages and within it I have it like init.py to make it into a module and then I have the list page page list page page object and the product page page object. So if I was to scrape another page say it was going to be a category page and it was going to return product links as well. I could simply go ahead and create category page in here add an item that matches it and then in my spider wherever it was appropriate return that data. So let me show you what comes out. I've just run this and we can see here that it returns us these items all neat as data classes ready to be exported. Scrapies feeder exporters work with data classes as well as atas and now pyantic as well. And then we can just export that however we need to or you can push this through your pipeline. Now I'm using the zite API for this. Uh zite API works extremely well with scrapey. It's what we use internally. In fact, we use Web Poet as well internally, Scrapy Poet internally um just to make our lives much much easier. And uh I mean I learned this from the scrapey maintainers as well and they all use it and praise it as much as they can. So I think it's definitely worth learning. It makes your code much more maintainable, structurable, testable, which is only ever going to be a good thing. And it says goodbye to these massive long unwieldy spiders with lots of passing. And it also gives you the opportunity to maybe write less code. One spider could handle multiple different sites if you wire it up correctly. I'll leave a link to the documentation so you can go ahead and have a look at that too. Um, I'll leave a git repo for this as well so you can see my code here and have a play around. And also let me know what you think. Leave some comments. What do you think about Scrapy Poet and Web Poet? Have you used it? Are you going to give it a go? What are your thoughts? Thanks a lot. See you in the next one.