Gemini 3.0 Pro Code Gen With Web Scraping Copilot

With the release of Gemini 3.0, I wanted to see how well it worked with web scraping copilot, which is our VS Code extension. My favorite model before this was Gemini 2.5 Pro. So, it made sense that I gave it a go and hopefully see that it's better. Some of my colleagues have already tried this and there's a blog post down below where they post their results. Want to show you mine here. Now, for those of you that don't know, O scraping copilot is our VS Code extension for scrapey. It helps you build out your scrapey spiders and writes all the passing logic for you, making your life much much easier. There'll be another video here linked that you can watch, too. So, I'm in web scraping mode. We're using Gemini 3 Pro. So, we're going to do create a page object using the item product item, which is an item I've created already for my schema, and the sample URL of this, which is from my test site. So what this is going to do is it should check the setup of my project which hopefully is fine because this is my demo like demoing project and then create a to-do list which it has here. So it's going to check the requirements confirm then create a blank page object. So we're using scrapey poet which is based on web poet which means we separate the passing logic from the extraction logic and we can better utilize uh copilot in this case with gemini 3 pro to actually build out that passing logic for us. Uh it makes it much neater and much more manageable and it also gives us the ability to run tests against it too. So you'll see that I'm going to generate a load of uh fixtures to test against out expected outputs that we can use and then also Gemini is going to generate the passing code or the selectors. So all I need to do is give it a URL. I don't even really need to check the page myself. So we have our fixtures. We had see them down here. We have our output.json which uh is expect which will have you know the the fields from your item and what you're expecting for them. And then the same in the actual page object itself. So we can see, you know, this is our empty page object with all of the fields and it's starting to figure out what it needs to build to actually get this information. There we go. It's popped straight up. This is the expected output. So this is what it will use to check against when it builds the scrap the passing code, which I think it's doing now. So far this has been pretty quick and pretty painless. So, I'm impressed, but you know, I wouldn't expect it to be any worse than 2.5. So, that's pretty good. Let's check this. We have some passing code in here, which we do. So, I'm going to hit keep, and I'm going to let it run the tests and make sure the tests pass first, which they do. And let's have a look at the code that it generated. So, what I'm looking for here is very, very concise, simple selectors that look robust and look like they would be something that I would choose to write. And so far, it absolutely looks like that. We see this is pretty standard. Pretty standard here. This too. Uh this looks pretty good. Uh we're using price passer which is our uh Python package to help you pass price code out of strings and stuff like that. So that's looking good. This is all looking fantastic. Yeah. So so far this has been this has been very good. A lot of the older LLMs that we started to use when we were building this out like GPT4.1 really struggled with generating this code even though we were giving it very very specific instructions through the extension. So this is really great to see. But obviously we want to um actually extract more than just one product. So we need the list page too. Now when I was going through this before I would suggest that you created a new chat for different page objects. So in this case, I'm just going to put it straight in here and we'll see how it gets on. So we'll say create a page object for the item list item uh using the sample URL of this. And obviously being my test site, I know I just need to go back to the the bare URL here. And that's got all the list of products on. So same process again. We should see that within this file, we get another page object. I'll collapse this one so we can see it. We should get another fixture as well for that page. Another set of output expected outputs. To-dos looks good. So yeah, we'll let this we'll let this get on and uh we'll see how it works. But so far seems pretty good. Now the adoption for scrapy poet which is crucial to this extension's functionality um has been pretty low u which is surprise to us but we think it's because people don't actually really know what it is and how to use it. So, um, I'd highly recommend that if you're interested in that, you get the extension and you give it a go because it's a great way to build projects out, see how it can help you, and see how splitting out the passing logic from the extraction logic is very beneficial. And obviously, you know, we can use it through the extension like this. So, so far this looks pretty good. Let's check the expected output. And that looks about right to me. Product URLs, next page URL, all good. So now we want to see selectors being built which we've got here. Let's keep that. Let's allow to run the tests and we have five passed. So let's check let's check the selectors for you know passing the actual product URLs on the next page. This looks pretty good to me. You know what I wanted was one sort of standard selector and that looks about right. I'm not entirely convinced we need URL join here but you know let's see if it works. It works. And the same for here. So this all looks pretty good to me. All of the selectors both through both of these page objects looks like something that I would write. And given that I know this is a website that I built, I know that this is a pretty good way of going about it. So what we're going to do now is come back over to our terminal and we'll run scrapey crawl products and we'll just check that this is actually working. It looks good to me. We're getting product information come back. So, I think there's about 115 products to go through on maybe three or four, five, six pages or something like that. Um, so yeah, this looks like it's all done a good job. So, so far, I mean, the the selectors, the code that it wrote was very good. I think it was better than 2.5 Pro, but again, that's kind of anecdotal. Um, the speed as well, I think it was slightly quicker than 2.50. I've only been recording for about six or seven minutes, so generally that would be pretty good for me. So that's probably the quickest I've ever the quickest end that I've done. So yeah, very pleased with this. So so far Gemini 3 Pro looks like a very very good option for code and code generation and especially you know looks good to work through our extension. So yeah, links in the description below. Check out the extension, check out the blog post and I'll see you guys in the next one. Bye-bye.