How I go from Zero to Data in 10 minutes with Scrapy

Ask any developer what the hardest part of web scraping is. They'll probably say getting blocked or managing proxies. But I think they're wrong. The real bottleneck, the place where 90% of your development time disappears, is this. It's the tedious, mind-numbing grunt work of passing messy HTML, manually inspecting elements, writing selectors, and then dreading the moment you have to write tests for it all. It's like being a world-class chef who has to spend all day just dicing onions. This prep work is what cost you. And today I'm going to show you how we automate almost all of it. I'm going to build a complete ready to go spider from a base template in just 10 minutes. That's including saving some output from our schema. And I don't just mean a simple script. I mean a robust spider and the complete pi test unit test to prove that it works. We're going to do it using our VS Code extension web scraping copilot which is available now in beta on the VS Code marketplace. The challenge has started. Let me know if you can do it faster. So, so far all I've been doing is I've just been checking over my base project, checking the set settings, making sure my scrapey poet is set up, making sure the spider's configured properly, and also making sure that my items are created. I'm very using a very universal and generic template here that I created because I've done a lot of testing using this extension. Now, I've set the agent to web scraping mode and I've chosen Gemini 2.5 Pro. In my experience, the better the model you can use, the better the output. And Gemini has been working very well for me. I'm going to prompt it and ask it to create a page object for the product item, generate fixtures, update code and expectations. So what this means is with scrapey poet which is built on web poet, we separate the passing logic in scrapey from the spider logic. So what that does is it gives us this new page object file where we have all of our passing code our se selectors based on our item all set up for you here. Now if you're not used to this which I suspect quite a lot of people aren't the extension is going to guide you through the setup required and tell you if anything's wrong to make sure that your project is ready to go. We can see here that it's given me seven steps that it's going to go through and one of them, the first one was checking scrapy poet setup. Obviously, my base template has it set up properly already. So, it's going to go straight in to creating an empty page object. As you can see here, this is it. This is the page object, and it has a field set for every field on our item class. And we can see that it's going to return an instance of our item. We have handle URLs as a decorator. So the spider knows which page object to send the data to when it receives some information back when it's scraping. The next thing that's happened here is that it's generated a fixture. Now what this is is it's a copy of the HTML of the page and an expected output.json file. This is going to be updated. All of this is going to be updated by the agent in the correct order. is going to figure out what the expected output is and then it's going to create selectors and pass in code within the page object to get that output and then we have the ability to test it with piest to make sure that it all matches. We can see here it's implementing the fields now from the MCP server. Now what this is is a locally run MCP server that comes with the extension. None of your data, none of your information gets sent back to Zite. This is all run locally. The great thing about doing it this way is that we can test against it and if something goes wrong further down the line, we can regenerate the effectures, regenerate the tests, and fix the selectors that aren't working anymore all through the AI. You'll notice at this point I haven't looked at the web page at all other than to grab a URL at the start and I don't intend on ever going to view source or inspect element. I just don't need to. So we can see we're running up to step six where it's saying we're going to run tests and it should have updated our page object which is just doing now. In this instance it's didn't work with the it didn't update the output JSON straight away but we can see here is the code that it's generated to pass. All looks pretty straightforward to me. Uh so what it's going to do now is it's going to say I didn't update the output.json. I'll do that now and then we'll run the tests and hopefully we get full test coverage. All pass tests here. So here's the expected output from that page. It takes all of the item fields and it's worked out through the through the MCB help what it should be and we get 13 tests passed and that's for each of the fields. Now all I'm going to do is repeat the process for the list item page. Just going to grab the URL here and I'm going to create a new chat u because I want it to be, you know, a new context and everything like that. And I'm going to set it off and I'm going to do exactly the same thing. Now I've built the spider in a way that it's going to want a list item to check through get the product uh product URL list and the next page for pageionation and then it's going to have its own URL as well just in case we need that. So again same thing create a page object for the item list item generate fixtures and update code and expectations. Here's the URL and it's going to run through exactly the same process. What this means is it becomes incredibly easy to take a universal spider that kind of and create multiple uh page objects for different websites through the same spider. So if we wanted to scrape two or three other sites that were similar to this one, this is my test site that were similar that met the same schema, so similar product sites or whatever, we could easily just carry on this process and say, "Here's some more sample URLs for this schema. Do this. Let's do this and it's going to create page objects based on the domain. So it would create a new file for a new domain and then we just run through it all and we'll scrape through it all as well because the page objects are are determined with the handle URLs. Like I said earlier, it knows where to go and it knows what item to return. So we can see here I'm on to step five. So I'm determinate determining the expectations. So it's working out what information should be being returned. I'm tidying up the handle URLs. Sometimes it doesn't quite get those right. So, I'm just going to tidy those up. That's not that big a deal for me. Now, all in all, this is going to take about 10 minutes. It's going to take about 8 minutes to generate all the code and everything and then 10 minutes to to run it and create some data as well. Now obviously whilst I'm using a base template here that I created I've do actually can do the same thing by I can do the same within the same amount of time even when I'm writing the code out myself because whilst it's generating the code for the page objects I can be updating the spider I can be changing settings or doing whatever I need. Now we can see it's generated a new fixture. Now when you when it generates the fixtures they go through your scrapey project. So, however your scrapey project is extracting the HTML from the pages or rather downloading the pages from the website and is the same way that it's going to be doing it here through the fixtures. So, if you're using the scrapey API plugin, it will use that to generate or if you're doing your own thing, it will use that to generate. You will come into a problem though if you can't actually access the page. You'll need to figure out your own antibband solution which you can absolutely do here or uses API. So, we can see that this is finished and it's going to update the expectations. This should be all of the product URLs. Yeah, it is. There we are. We should have a next page URL. Brilliant. So, that's about 8 minutes and we're just about done already. Now, it's going to update the actual um selectors and passing logic here. Uh, and it should be uh about to run the test. And then we'll just know whether we're all done, if we have any issues, which we don't. passed all of those tests. So all I do now is I just go straight away and say scrapey crawl and get the output into a JSON file for all of these products. I think there's aboutund and something on this test site. I actually left a download delay on this which I didn't mean to which is why it's going to take a little bit longer. But essentially within 10 minutes we've created uh all of the passing logic we need. we've created up from our template spider and we are running our spider for this website within 10 minutes. Now, we're obviously still working on this. This is the beta release. Uh but we feel like it's in a very good place. So, we really want you to get out there and try it and report back to us with any issues that you find and anything that you want to suggest. It is free to use and it's not tied to our services in any way, although it does work with the Zite API very well and integrate in very well. If you don't want to use that, but you want to use the web scraping copilot, it's not integr. It's not mandatory. So, you can go ahead and use it. So, I'm just going to let this finish up and it's going to take about 30 more seconds to download the rest of the data. So, yeah, go over to the extensions tab in VS Code and search for web scraping copilot or you can find it on the marketplace, too, and check it out. Download it and give it a go and let us know how you get on.