How I use AI and MCP to Scrape Data

Coding with AI and Copilot is great, but often we want to give it access to specific tools like APIs that we have access to. And that's exactly what I'm going to show you in this video. I'm going to build an MCP server using Python and fast MCP. And we're going to give it access to the Zeit API, which is going to give it a load of powerful web scraping abilities. Now, here at Zite, we scrape loads of data from loads of companies, millions and millions and millions of rows. So, if you're interested in web scraping, there'll be some links in the description below where you can try our API and uh I'm sure you'll be happy with it. So, what I've got here is my project file open and I have my very simple fast MCP server here. This is from their demo and you can just copy this out. But the key thing here is this MCP.tool and this is a Python function that's going to do something that your AI through Copilot in this case is going to have access to. Now what we want to do is we want to create av file uh folder and then add in this mcp.json file. Now this is how to do it per project right so this is only going to run in this project. You can install these globally but I prefer just to keep them within my project. So I want you know if I'm doing a web scraping project I can put this in and it has access to my web scraping tools for example. Now this is fairly straightforward JSON. You can actually generate this with fastmcp install mcp.json and then main.py and I've done d-copy that copies most of this to your clipboard. You'll need to add in the servers and then you just paste everything down here with the uh location of the file that you want to run that is your MTP server. So what you can do now is you could take this file and copy it into any of your other projects and your co-pilot and your AI would have access to it within that within that project. And now to check that I'm going to do start and we should have no errors. We can see that it's running. You can also do command shift X. And we can see down here that there's this demo MCP server. This is what I've installed here. You can see by the name and it is indeed running. You can also install it uninstall it from here as well. The other thing we can do is open up our agent mode and down in the bottom right probably behind my head there's a configure tools button and we have here we can see we have the Pance MCP server which is fine. And we have our demo MCP that has add two numbers together which is exactly what was written in our main file here. Now we have access to this. So let's say call our co-pilot and say uh add five and six. And I'm going to tell it to use the MP MCP server because otherwise it will just do it by itself because obviously these are not the sort of tools that you need to worry about running. We can see here it's coming up here. It's saying demo MCP input and it's going to run this. I'm going to say uh allow uh in this session or just click continue and it's going to come back with the answer. This is run through that tool that was created in our MCP server here. So what does this mean? Well, we can effectively put whatever we want in here to then have access to that and give our AI that. Now, in this use case, what I'm going to do is I'm going to create a tool within the server that has access to the ZI API. So, it's going to be able to access data from websites much more easily. It's going to download the HTML and then I can use it to generate CSS selectors or XPath in a much better way than you could do normally. Now, this is obviously quite useful or you could use it to do the auto extract and give it product information. So, you could do something else. So, what we're going to do is we're going to go ahead and create a new tool. So we do at MCP tool uh extract HTML and this is going to take in a URL which is a string and then it's going to return a dictionary in this instance. So now to do this I'm going to need to use requests. So I'm just going to do import requests and I'm going to come over now to we'll just put pass in here so we forget. I'm going to come over now to my Zite playground like this. Let's make this a bit bigger. And we're going to go ahead and go grab code snippet because I want the HTML like this. And this is going to give me the code snippet that I need. So, I'm just going to copy out this part. We don't need that, but I'm going to copy this. We're going to need to remember that we need to import B 64 because we return the HTML to B 64 encoded. Let's come back to our Visual Studio Code and let's paste this in. Make sure this all lines up. And now we can go ahead and let's just change what we need. So we need to import B 64D like so. Thank you. Copilot done. Now I'm going to return and it's going to be a dictionary. So I'm going to call it HTML and I'm going to say that we are returning the HTTP response body. Let's move this in. Now we need to now we need to have our API key. There's two different ways you can do this. Obviously, you don't want to be putting your API key directly in, but what you can do is you can put it into your MCP server or you can run it through the environment variables. For this demo, I'm just going to import OS. that I'm going to bring it in myself in this way through my environment because I know that if I'm running this MCP server within the environment that is going to be in my web scraping project, it's going to have my API key in. There's loads of information on the on the fast MCP of how you want to handle your environment variable. So, it's up to you. So, I'm going to call this API key and this is going to be OS.getv get amvi key and I'm going to do if a if API key is none. Thank you. We'll raise a value error. Not really what I want, but we'll just raise a bare exception like so. So now that you know if it doesn't find it, we're going to have an exception which means there'll be an issue. And this should probably go at the top of our file because if it doesn't find an API key, we want to not uh run the server and then we'll be able to see what they what the issue is. So I can get rid of it here. Okay. So we can change this up now. Uh your API key. So this is going to be API key and the URL is going to be the URL that we are asking for like so only follow redirect but we did want HTTP response body. So we should be able to save this now and let's rerun our server. So, I'm just going to come over to democp and I'm just going to go restart and it's going to say discovered tools and it says tool extract HTML does not have a description. We definitely want a description. So, we're going to go ahead and put that in here and it's going to say extract HTML content from a URL using that API. Perfect. So, tools must be accurately described. So, let's go ahead and restart now. Discovered two tools. So if I then come over here and look in this, our demo MCP server now has extract HTML content. So what I can do is if I grab a URL, let's just grab this one. Come over to my code and go extract the HTML from here using zite. Oops, spelled company name wrong. Shall I never mind. And it says, do you want to use your MCP server? Uh, yes, I do. And obviously you can make it so it does always allows in this session. But we'll just hit continue so we know that it's working. And now it's been successfully extracted. So I'll just say uh what is the title of the page? Just so we kind of get an idea that we that it knows what we're looking at here. Perfect. Now what I could do is I could say for example, let's say we were trying to extract, you know, something from this and our general AI wouldn't have access to it because it would be blocked. Whereas the site API allows you to mitigate that. We could say, you know, give me selectors for all links that contain the word what's on this page core. Let's put that in there. Let's see what we get back. And you can get the idea. So if this was a product page, for example, let's do that. Actually, let's let's do that. So, we kind of can really understand what we're looking at. Let's just grab a URL for a page here. And we'll do it again. We'll say uh extract the HTML from this page using sites. And I'll spell our company correctly this time. So, this is kind of like one of the things that I've been using this for because we're giving our AI access to pretty much any page that we want rather than it being restricted. You know, it comes up and says I can't access that for whatever reason. But now through that API and our MCP server we can. So we can do this and it will get it to extract the HTML from here. And I'll say this is a product page. Give me selectors for a standard product schema that work this page and uh return a code block using BS4 to pass it. Let's see what happens. So we can see that it's probably mostly correctly going to extract the selectors properly because it can see the HTML. So the biggest problem when you're trying to do this without site API is that it can't actually get access to the AP the actual proper HTML because it's been blocked. Whereas in this case it does and it's written the passing code for me. So that's my kind of example, my reasoning for doing this. It's very very simple to build these MCP servers. Wow, my screen is a mess. Let's close all this stuff up. Um, it's very simple to build these and you can then give access to, you know, your AI through C-pilot or Claude or whatever you use. They both have ways to install and run. You can give it access to these specific highly specialized tools which, you know, is very very powerful thing and that was, you know, very quick and easy to write. So, hopefully you've enjoyed this video. Um, please like, comment, and subscribe. I really appreciate that. There'll be links in the description down below to check out Zite API for yourself. is very powerful one. It cover all your web scraping needs. So, thank you very much.