Introducing Web Scraping Copilot for VS Code.

I want to share with you a project that we've been working on here at Zite and that's our VS Code extension. It's called Web Scraping Copilot and you can go ahead and get it from the marketplace on VS Code right now. Install it and give it a go and we would really value your feedback. What it does is it helps you build your Scrapey projects out by uh being able to generate parsing logic for you in your Scrapy PO page objects. If you're not used to page objects, I'll explain it all as I go along, but I want to show you how I've been using it and give you an idea of how you can use it and, you know, see how you get on. So, I have a blank folder here and I'm just going to create my project. Now, I'm using scrapey scrapey poet, as I said. I'm also using a few others. I'm going to be utilizing the site API for this and also using price parser, which is an open source um package that we've created, which you can use and kind of does what it says. So now I have this uh created. I'm going to activate my virtual environment and then I'm going to start my scrapey project. Now the extension will work with new scrapey projects and it it requires the page objects to run because when it generates the code, it generates a page object for you. I'll show you all of that. Now I'm going to do scrapey gen spider to get a simple spider up and running. Uh and I'm going to change a few settings. Before I do that, I want to explain that with the extension that we've built, it runs its own MCP server locally and it has a lot of instructions in there to help you with the setup of your scrapy project, your scrape poet project. Now, if you don't have it installed, for example, it will tell you if you don't have the settings set up correctly, it will tell you. However, I've used this quite a lot now, so I do this manually. So, there's a few different ways you can go about this. First thing I'm going to do is create a new folder called pages, which is where my page objects are going to live. And I'm going to create my init.py to make this a Python module. So I can import in and out from it. Now I'm going to come to my settings. I'm going to change a few things here. Add-ons we're going to need. It's scrapey poet add-on. This is going to do all of the configuration for us. And as I said earlier, I'm using scrapey zite API as well. It's up to you if you want to use this or not. The extension is not dependent on it. Then we need our scrapey poet discover. And this is going to be a list and it's our base pages that we just created. Base dot pages. So this is essentially set the setup for these add-ons that we need. Now I'm going to change a few of the other basic scrapey project settings. Now you can prompt the agent to help you build out your item if you want to. I prefer to create my item myself and then go from there with the uh with the web scraping co-pilot. So I'm going to import at which is my preferred method for um scrapey items. Now and we have some information here. I'm going to remove uh stock but we will keep the rest there. So I have my class. Now what I can do is start to prompt the agent with RMCP and instructions to help build out the rest of this project. Now I'm not even going to need to go and have a look at the website. It's going to do it all for you. I'll show you what I mean when I get going. What I'm going to do is just grab a URL whilst we're here. Right. So, let's get started over here. Let's move this a bit over. Now, what I'm going to do is change it from agent to web scraping, which is a new mode that we have. I'm going to select what model I want to use. I've had good success with GPT5 Mini. Um, but the best ones so far for me are the higherend ones. I'm going to use Gemini Pro here. That's worked quite well for me. However, whatever you want to do though, you can in you can use your own LLMs. If you have an API key, there's a way to set that up. Or if you want to, you know, I'm using a premium model here. As I said, it works well with GPT5 Mini as well. So, let's start uh prompting our way to success. So, let's say uh create a page object for the item product item and create uh and I'll just start there. Now, this should give it a to-do list. Um, and it's going to check the status of everything. So, when I talked about scrapey po at the start when I said, you know, it needs to make sure that you're all set up, it's done that here. It's checked that here. So, you know, you have the pi test installed. It's all set up exactly. So, I'm going to give it some URLs. Here's a sample URL. I'm only going to give it one. You should really ideally give it two or three. So, I've given it the URL and it's saying it's going to use our item, which is correct. Do we want to do it create the page object in our pages directory? We do. So I'm going to hit yes. This is going to go ahead and like download the URL, download the HTML from the URL, save it into the fixtures that we can then use PIEST to test our scraping logic, our passing logic against. So we So we've got an empty page object. I'm just going to have a quick look at this here. So what this is is the page object and it has a class with the handle URLs. It's returning our item and it has the fields from our item that we set up here ready to be populated. And it's this that the AI is going to create and fill in for us which is going to pass all the information. Now it's saying should I create uh create a test fixture for sample URL? Yes, we want that. That's how we're going to work through and generate the code that we need. So we can see it's come through uh under my general scrapey project. So however you set your scrapey project up to download the um HTML from the websites that you're scraping, it will use that to do so. Like I said, I'm using the Zite API, so it's all through there. Now, it's saying, do we want to proceed with updating the test picture with the expected data and implement the extraction logic, which is exactly what we want. I'm going to hit yes. It's going to take a little bit of time now to work through this. I found that depending on what model you use, it's quicker and you get better results. It's going to prompt me to say we want to use the MCP server. Now, of course, you can allow it to use this without having to have the interruption, but for me, I like to have it there just as a break point. So, when I'm explaining what I'm doing, I can see it come up and I can tell. So, if we click on this, we can see it's taken our input uh and it's determining the expectation. So, I'm going to come back over to the fixtures. And if we look at the input, and I move this out the way a little bit, and we'll see this is the actual HTML for the page that I gave it, sample URL. And the output here is the output from our item and the page object. They're all linked and at the moment they're all null. So through the MCP server which is run locally which comes with the extension. I can actually show you the um the tooling here. Here we have update page code expectations and you can read through this and it explains uh what it all does and the instructions for the um for the agent there. So, this is going to go ahead and um figure out what information should be here. And we've done it now. So, we're going to run tests. Uh yes. So, it's going to update the output. So, we'll should see the correct product information appear now here. And then it's going to use that to work out the selectors, update the and update the uh page object. So, let's let this run. There we go. That looks good to me. We can we can uh clarify that. That's fine. And we also have the updated page object. You see it's using price passer. It knows that it's there so we can use it. I'm going to hit keep on all of this. Have a quick look through. Looks good to me. I'm going hit allow. And we're going to let our tests run. We can see that all of our tests passed. Now, let's make this work properly like a spider. So, I'm going to go to my spider, go to products, and we'll go ahead and get rid of all of this because we don't need this. We want to say that our URL which is going to be a string in this case and paste that in here. That's our start URL and we want to do async defaf because we are using modern scrapey which is async and a wait and the start here. We want to yield the request and our callback needs to be self.pass and I'm going to call this pass book. I'm going to import in my product item from up here. So we'll do from base do items. We're going to import in our product item. And now we can have our async def pass book. We don't want that. We want it to look like this. We want to pass this in here. So then we can say our item is that. So we can yield our item out. So very very simple spider just to do this. And then let's run scrapey crawl products and run our spider. Just going to run away. And we should down here somewhere have the information. Here we are. You can see product item here and this will match what actually was in our page object what was in the output JSON and you know we've scraped this directly from the page. We ran our tests we know that they are passing. Now obviously more URLs that use a sample more tests you create the more accurate the air the agent can be in creating the passing logic. But you can see right there that I didn't have to look at any HTML myself. I didn't have to go through and find any HTML selectors or anything like that. It was all done automatically using our extension as a guide through for the agent. So, let's go ahead and create another one because, you know, we don't just scrape one page. We're going to scrape multiple. So, I'm going to create a new item and this is going to be a category item. There will be a um book urls which will be a list then a next page URL which will be a string. So now I've created this. What I want to do is I want to have a new page object for it. Now generally speaking I recommend uh creating a new chat when we're doing this because we want it to you know um concentrate on this and not worry about the other page object. So I'm going to say create a new page object for the item category item uh in a separate pi file in the pages directory. Use the URL. Oh, and I need to grab my other one here. use the URL as a sample. Create fixtures and update code. So, I'm going to try and see if I can get it to just do as much in one go as possible. So, you know, it might be a bit quicker for us. Uh, again, it's going to check the status because obviously we're starting it process from the beginning again, but you it should blitz blitz through this nice and quickly. I'm going to update my spider whilst it's doing this. So we'll import our category item. Then instead of pass book here, we'll have pass category. And we need to uh create that here. Not like that. Thank you. Oh, can see now it's creating the fixtures, which looks good to me. Uh async defaf pass category. Like so. Right, that looks about right to me. I'm going allow it to use the MCP server. This isn't right though. I need the response here because I don't want to re yield a new request. I want to do responsefollow allow sorry because you know that's going to make sure that the URLs are formatted correctly because it knows what the page looks like. There we are. Cool. So now I've set my spider up and you can clearly see how easy it is to make quite a generic spider. If we were scraping lots of different book websites, they would still come under the product item and the category item, right? So, they would still fit. And as long as your schema fits into those items, which I'm assuming it would do if you, you know, want all the data from those websites to be together and formatted properly, we would just need to create new page objects for each of those sites that fit into that. And with the um extension here doing all of that work for you, it's very easy to create multiple page objects with one spider and make your workflow much much quicker. And again, we can run the tests whenever we like to make sure that we're still testing properly. Well, we could regenerate the fixtures if we think the website has changed. So, we could go ahead and do that anytime you get an error or you want to rework your selectors and get it to redo it all for you. and then update the tests, update the output, update the selectors and then carry on running your code without any manual intervention from you other than you know asking it to recheck and redo this redo the selectors and the fixtures. So that's the real benefit of having it like this having all the logic split out and the ability to do that with pi test. I'm going to click keep. So uh keep that as well. And it looks like we've updated our thing. Now, it didn't create this in a new file for me, unfortunately, even though I did ask that. So, that's one thing that we would want to do slightly differently. It's best practice to keep your page object classes in separate files. You see, we're still passing all of our tests here. So, I without actually, you know, looking at anything, I'm just going to clear this up and I'm going to run it again and I'm going to do O for my output.json. And we should now Oh, I didn't change the uh URL. There you go. That's my fault. So, we need to actually update the URL that our spider's starting from because, you know, otherwise it's not going to find what it's looking for. Okay. And let's give that a go. We see it's found loads of different pages. It's zipping through. And if I come to my output, that was my failed attempt, which is why it looks like that. I should have 30 something books here with their price, skew, and the information that I've selected. Again, I've done all of this. I haven't written any passing code myself, but I've remained in control in my environment. I've remained in control with the of the code. If I didn't like the look of any of this passing code, I could get it to regenerate it or remove pieces that I don't want. I remain in control. I have code that I can test. You know, I can test this with my with Piest. I can test it to make sure that it still works. I can regenerate the fixtures with sample URLs anytime I like. Just prompt the agent to do so and it will get you those. and go from there. So, you can see how easily this can be turned into something that can save you so much time when you're scraping data like this. So, please go ahead and come on VS Code, download the extension, give us feedback. There's a link to the GitHub issues where you can provide direct issue feedback for us and let me know what you think. You know, if you have if you give it a go, write a comment under this video. I'll monitor it or reach out to uh reach out to us on on LinkedIn or anywhere you like anywhere you can find Z or myself and let us know how you get on and what you