Generate HTML Parsing code the right way with Scrapy & Web Scraping Copilot

One of the areas that we can absolutely make our lives easier with AI and web scraping is by writing passing logic code. This is a very laborious and tedious task when you have to go through pages and pages of HTML. So, it makes sense to be able to utilize the AI's capabilities here. However, if you've tried this and you've just dumped a load of HTML into an LLM, you'll know that it's not very good at doing it and it will quickly tie itself up. Now, our solution to this was a VS Code extension designed exclusively for writing Scrapey code. Within Scrapey, there's an add-on called Scrapey Poet, which is based around the page object system. I'll explain that as we go a bit further, but it's very very crucial and integral to the act the actual usage of this extension. It allows us to separate the passing logic from the crawling logic within Scrapey. So, our spiders no longer have that passing code within them. that's handed off to a separate file, a separate page object. And then within that, we can get the LLM through C-pilot to write that code for us and it will test against it. It will download the fixtures and we can actually start to build up something that is robust. And when it fails, we can run it again and generate more parsing code. So, what I'm going to do is I'm going to run you through a complete scraping project here where we're going to scrape some data from a website utilizing web scraping copilot and the ZI API. Now, I'm going to note here that although we've developed this and it isn't open source, it is free to use and it will be free to use. It integrates very well with our product Z API, but it's not mandatory. You're still using with your Scrapy project. So, however you choose to actually ex uh download the data from the website, from the server through scrapey is entirely up to you. Web scraping co-pilot will still help you, but hopefully you'll see and understand as we go along. Now, if you haven't got the extension installed, you'll need to go to the marketplace and get it installed. If I look over here in my VS Code, I've got it down here, web scraping copilot, and it is installed at the moment. There's a few very important requirements that you will need to set. The first one is to install UV. UV is a more modern Python way of installing packages. Now, in future versions of the extension, we may remove this requirement and make it work through pip, but at the moment, this is required. So, just check when you're installing the extension that you have this installed. Uh, it's very easy to install on Mac. I think you can just brew install it, for example. The other thing is these two chat MCB access commands within the extension itself. It brings its own MCP server that it utilizes to actually run some of the instructions and show the uh LLM uh co-pilot in this case how to actually do it. But this is a local MCP. No data comes back to us. So once you've got through this, we'll come back to this in a second. You'll need to come down to your chat window and start selecting a model. Now we've got a post that we've just released today actually that talks about the best models to use. Now, obviously, this moves very quickly. I prefer Gemini 3 or GPT 5.3 codeex at the moment, but check that link in the description to see what we're recommending at the moment. I'm going to leave on GBT 5.3 codeex. The next thing we need to make do, make sure we are is on web scraping mode here, which we can select here. And now we are good to go. So, I'm going to close this up. I have some URLs saved that we're going to be scraping from. And I'm going to go ahead and start my project. This is just a blank folder. And I'm going to come into my terminal. So I'm going to do UV init my project folder. That's going to start a project for me. And then I'm going to do UV add. Now I'm adding quite a lot of things in here. Scrapey is obviously mandatory. So is scrapey poet. Piest is required too. The other ones uh I'm using because I'm using the site API. So I want to use scrapey zite API. Again not mandatory. Price parser is an open- source uh package that we maintain which is just helpful at extracting prices from within uh HTML etc. and the extension and the LLM knows how to use this, so it's worth installing. Let's get all of this installed and then I'm going to activate my environment here and we should be good to go. So, I'm just going to click on a file and hopefully it will pick up my interpreter down here within my within VS Code. And now I can do a scrapey start project. I just call my project base and I want to do the dot to put it in this folder. And now we can create a spider. So, scrapey gen spider products and the URL. So, that's been created. So, at the moment, this is a very simple basic scrapey project which we can see if I was to go into any of these things here. This is just fairly standard. Now, this may change going forward, but at the moment, you need to get to this point before the extension will start to help you. The extension is designed to help you build passing code through page objects, but it will also help you with some of that setup, too. And I'll show you that now. So, from here on out, hopefully I won't need to write any code myself. It will just be done by Copilot through our instructions. Now, obviously, I'm using Copilot, so you will need a uh C-pilot license. I have just the standard $10 a month one here, and I'm using a one times model. You can obviously use some of the free ones, but you know, they might not be as good. So, what we're going to do is we're going to come over and we're going to open the extension panel, and we're just going to do check again. and we'll do setup environment. Okay, so we've got that and we'll just make sure that we have the right interpreter selected. And now we can see that we are setup complete. If any of these are not checked for you, you need to make sure that you address that before you actually go ahead and build out anything further. Now we can see here that it's saying your environment is ready. Generate some passing code. Now what I want to do first is create an item. So we can see when we go to items, it says no items found. Now the item is obviously the data that we're going to scrape into. You can do this yourself manually. You can use predefined items through Zeke common items if you wanted to or you can get the uh the agent to write it for you. Now this is through the chat through your just standard kind of co-pilot here. Although I do have the web scraping selected down here. So I'm just going to say uh create a product item using atas uh with the fields price, description, name, and skew. So I prefer to use atas with my scrapey projects. However, you can use whichever one you want. If you want to use scrapey base items, that's fine. Data classes also fine or pantic if you wanted to. I just like ATS and it's built in. It's in built in with the scrapey as well. So you don't have to install anything else. So I'm just going to let the uh co-pilot here go ahead and build this item for me. And we'll just double check it. And then we should be able to push on through this and click this button here by generating passing code. So I'm going to open this up. Check my item. I don't want you. Thank you. I don't need any of these. Don't need this base item. We'll get rid of that. Get rid of that. And we don't need this part up here. So, we should have price, description, name, and skew. Fantastic. I do want to generate one more item because I want the actual product that I'm scraping, but I also need to scrape all of the items off of a page, and I want to put that into an into an item as well. So, the name page object kind of helps you a little bit here. So, each page has its own defined object, and its own object needs an item to work with. So, you'll see what I mean in just a second. create another item called uh uh list page uh with the fields URL product urls and next page URL. So the idea being here is that when we go to one of those product list pages, we have this item class available that we can then use to populate the data into uh rather than just doing it manually like you might have seen in some of the old school scrapy tutorials. This is very much the modern way to do it and you'll you'll reap the benefits so quickly. So we have our two items, product item and list page. I should probably call that list page item, but that's fine. We'll carry on. Now I don't actually need this chat anymore. Uh we'll just move it out the way. And what we're going to do is we're going to come back over to the extension. Hit refresh. And we can see we have two items and it shows us the fields here. And if you clicked on these, it will take you to that point where in your code to see. But we're not going to need that now. We're going to come to page objects. The next one down. I'm going to do generate passing code. And this is going to open up this new new LLM chat window here. Just make this a bit smaller. Pull this down. And I'm hoping now that I don't need to leave this until my project is complete. And all I'm going to do come come down here and write, you know, run spider type thing. So this is telling us what we're doing. It's missing configuration. Yes, we do want to update the configuration. Once you get used to this and you understand what you need, you'll save so much time because you'll just be able to go into your own project settings and just change all what you need manually. or maybe you've got a template for your scrapy project that you use. So you don't need to worry about this. But I just want to show you how much the extension will help the copilot the LM to actually go through it. Okay. Proceed to step two. Yes. So it's going to say it's reading sample URLs. Uh three product pages. Yes. Use these for step two. Now I put these in a file just to save them. But if you don't have this file, that's fine. It will just ask you here, can you give me some URLs? And you just paste them in the chat. I just did it ahead of time, so you don't have to watch me go ahead copying and pasting things backwards and forwards. So, we can see over here, we now have a page object over here. And it's starting to build it out. And we can see that we have the fields. Now, I'll show you the code for this uh at the end because, you know, right now we don't need to worry so much. So, now that we want to proceed to ne to the next step. Now, this will probably fail. But this is where I'm going to talk a little bit about the Zite API integration. So, it's going to talk to you about browser response here. This is very specific to the um ZI API add-on. So, we can see down here that it's failing. It's not scraping anything. So, we can see that we got 403. Choose how to retry. So, if you come across this, you need to change your settings to make sure that you know how you can access the page with through scrapey. But if you were to use the ZI API, you can come over here and do this part here and you will say checking for Z API support. I have my ZI API pro uh API key stored as an environment variable in my terminal. So all I need to do is click this button and it's going to run the setup tool for that. So again, this is an optional step, but it makes the whole thing much much easier and using our Z API, you don't get any blocks, bans, it avoids all of that stuff for you. Uh, and yes, please put that put that line into my settings. Just let it do this. Cool. Perfect. So, we don't want to do any of that. We should now, if we refresh this project is using that API. Perfect. Now, I'm going to come back here and I'm going to look at this. What was I step four? So I'm going to do uh retry step four. Same urls. If we come to our page objects, we should start to see down here now that this is going to run again and we will have access to that page and we should download the HTML from that page that comes back from the Zite API and we can then use that to work with with our page object. So what it does is it's going to download the page the HTML that it's that your scrapey project would see. It's going to save it into the fixtures, the test fixtures. From there we can then start to let me see it just hit yes. From there what we can do is we can use the instructions that we've given to Copilot to actually look through and find the information on that page. It's going to find the information on that page and it's going to generate selectors and then we'll have piest tests to run against the code that's generated versus the expected output from those selectors. So we'll know right away based on the test that we've got whether the code that copilot has written with help from our extensions instructions is working. Now, if it isn't, you can retry or, you know, it'll give you a couple of options to retry or you can go and then and check yourself and make sure that you it's got the right selectors. Generally speaking, it's pretty good at it. But, of course, one thing to note is that the more test pages you give it, the better it will be at getting the data. So, we recommend at least three. Although, if all the pages are really similar, you can get by with one. But, it doesn't take any time really. So, I don't know why you wouldn't just do more. So, I'm just going to let this run. It's going to generate the code and check the fields. It will generate the output text uh the output.json sorry that it's expecting and then we can compare that. We can compare the selectors and we can run the tests. And if we see all tests pass, we don't need to intervene. So it's read through it and it's generated what it thinks it's going to do and now it's going to implement that based on the instructions that's come back from the MCP server. So, it's going to update everything and then it's going to write that code. We can see it's just doing that now here. So, I'm going hit keep on this and I'm going to say proceed to step six. Yes, please. And this should be I think the test. Oh, no. We still the step six is generating the actual code. Here we go. I was right. It was the tests. it just uh couldn't find the installation. So let's have a quick check. So we can see that all through through each of one of our tests 1 2 3 uh we've passed everything. 18 tests passed. So we are good. Now it's going to say do you want to integrate this page object into products? Now products is the spider that we created. But actually what I want to do is I want to generate a page object for the second item. So, I'm going to say generate a new page object for the list page item using the URL of this. So, we want to make sure that we've got both of our page objects done and the code written and tested before we build the spider in case anything needs to change. Now, obviously, when you're doing it like this, the uh you'll have multiple page objects per domain and they'll tend to go into the same file. But when you do this, if you have multiple sites that are essentially the same but very very slightly different like you know multiple e-commerce stores, you can actually create page objects for them within one spider. So because the page objects may make it more generic, they can have different ways to manage it and they'll be able to check against the URLs as to which page object to use. So basically it makes your life much much easier. So this process that we're going through here, we can do just a part of it again and just give it different URLs for a different domain and we can start to build out a spider that will be able to scrape multiple sites very very easily. So it's just zipping through creating a new page object. This one's much easier because there's only one sample URL and um only a few actual pieces of information. So we can see here the fields. That's refreshing because I'm just putting in the test fixtures. You see test fixtures here. Okay, created the fixture. So, let's just carry on. Yes. So, notice I haven't looked at any code yet. I got my passing test. So, I'm just going to make sure this test passed and then we'll just push on. We'll have a look at the code at the end. But, you know, obviously generally as you go through, you want to check the code that's been generated. I just want to give you an example of what can be achieved. Okay. So, that's worked its way through that. And it's generating the output JSON. Now, this is going to be larger because and you can see up here it said it's generated a file due to size and that's because my expected output here is going to be a very long list of URLs. I think it's as many as you know 50 60 products on this page. So, it's got to go through and it's got to make sure that they're all there and make sure that the selectors that are going to be generated get those out. So, this is one place where it does get a little bit slow, but it needs to work out and it needs to know what it's expecting to see so we can actually test against the selectors. If we can't do that then obviously you know we can't validate how good the code is or whether the code is working. So I'm going to proceed to step six. So we did have one fail. We can see down here somewhere that we are getting a lot of URLs, but we missed out on some of these for some reason. Now, generally speaking, um the selector will catch all the URLs. This is probably because there's so many. Um but this is something that I want to look into. Now, obviously, we would just need to have a quick check through, but I'm going to carry on. I'm going to say um uh update the products spider with these page objects. And we'll check the output when we run it. And we'll just make sure that it's working. Now, I'm trying to do this as hands off as possible, but this is more this is definitely designed to be a helper and save you time. It's not obviously supposed to be completely handsoff. So, in other demos that I've done, I've actually written all the spider code myself. I've checked over through everything. I've tweaked selectors and stuff like that. In this one, I just want to see how well it will work by just letting Copilot do it all. And so far, it seems to be pretty good. But, you know, we'll run it in just a minute and we'll see what the output is. And we'll see whether we need to update the selector that gets these URLs because some are missing. So, we want to run a short crawl. Uh, let's just say yes and see what we get to. Um, it usually has an issue with async and await. Modern scrapey uses async and await and sometimes the co-pilot gets a bit confused as to what to do with it. So, let's just let it run through and see what we get to. And we saw it just fl up there. It said something was not not awaited. Yeah, here we go. It's going to update them for async and await. I don't know if this will work. We'll hit okay. If this doesn't work, we're going to just run it manually anyway. Let's leave that alone and come back over here and hit run. And we'll just do dash o output.json l and we'll do run spider. Okay. So we can see that we are getting products. Now, originally it uh doesn't do the async part and it doesn't do this properly. Um this is not the best in my opinion. This is not how I would write this spider. However, as you can see, it is working here. And the reason why it is perhaps a little bit slower than what you'd expect is because we didn't change any of the settings. So, if I was to come back to my project and go to my scrapey settings over here, you'll see that we still have um we have no yeah, we still have concurrent requests as one. So, that's why this is going to this is running so slow. We could easily update this. What I wanted to show with this is the fact that the extension that we've written is for scrapey projects. It's for page objects and it's designed specifically to write passing logic for you. I've gone further here to try to get it to do much much more. However, that is whilst it's capable of doing that, it is leaning on the LLM, the copilot much more. Ideally, what you would do is what I've done in this video where I show you where I write most of the code myself, but I actually just use the extension via web scraping copilot to write the page object code and that is much better and much much quicker. However, if you've never used page objects before within Scrapey, this is a great way to start trying. If you've never used Scrapey before, I've proven to you here that just by prompting the OM and following along with what I did, I was able to get a working spider going. So, what we would really love for you to do is to download the extension and give it a go and try and get this far. get choose a website and get as far as I have here and get it working and then come back to us with feedback and let us know what you think. Now, obviously this is still pre-release. So, it's still in beta. So, it may change, some things may change, the functionality will remain the same or slightly improve as we iterate over through it, but we might update the UI. We might update it to make it easier for you to access, for you to actually work with. So, make sure you keep it up to date and yeah, give it a go and let us know what you think.