Modern Web Scraping starts with THIS.

One of the most common mistakes I see people make when they start to scrape data is that they think that they can just use the tools that they know and love over and over and you'll get the same results. So I want to show you that you don't actually start with firing up your code editor and installing requests and beautiful soup. You actually want to start with the website that you want to get the data from to really understand what's going on and what you need to do to achieve your goals and get the information. So on the screen I have my test site here and this is just a site with loads of different products on but there's a few key things that we're going to get into. So I'm going to open up inspects and I'm going to go to network and I'm going to select fetch/xhr and we're going to refresh the page. Now if you've noticed there we had a browser check and I'm going to come to that in just a minute but the key thing that I want to show you is that we have this fetch request here which is a request to the API that the site uses to get its data. Now, most modern websites work this way. There's a front end and a back end and they request and send data in JSON format between the two so that the front end can hydrate it and make it look the way that it needs to be for the for the end user. So, this request here has a response and it has our API and it has all the products in it from this page. Now, this is pretty cool because everything is already in JSON and it has all of the information in it. So, we don't need to do any passing or anything like that. we can just simply mimic this request to and get that data. But how do we go about doing that? So, if we look at headers, we can see that there's quite a lot going on here. And in a lot of cases, there will be even more, but there's a few that stand out straight away to me. And the main one is this one, this cookie token here. Now, that's going to be important, and we'll come to that in just a second. So, what you want to do when you actually find this, assuming that it does exist for the site that you're scraping, you want to go ahead and come and copy this as curl. Now, this is going to pull all of the information out, all of the headers and the cookie and everything, and we're going to put it in this case into Bruno. You can, of course, use Postman or Insomnia or whichever one you want to use. So, I'm going to go and create a new request from curl. I paste it in here. And I'm going to hit create and give it a name. So, we have everything in here now. We have the parameters and the headers we can see here. And if I run this request, we should indeed get the information back that we were expecting, which we do. Looks good here. Now, there's two things I like to do first. And one of them is to just mess around with the parameters and see what we can get to. So, we can hit we can do page two. That does work. And three, and so on. Great. So, we know that these this is good for us. This will work. But now we really want to think about is how can I actually um mimic this request programmatically so I don't have to use the browser and I don't have to go through it that way. Well, there's a few things that are going on here to be able to do this. We know that we're probably going to need a good or token and cookie. So, I'm going to remove that. And you can see right away that this is not going to give us what we want. We have the uh response from the website basically telling us that we're blocked. This is my hint here. So, uh, this is my test website. They won't tell you this. Normally, it will just be a 403. So, I'm going to add that back in. And we're still good to go. Now, we can take out other ones. So, let's take out uh let's take everything out and give ourselves a blank request. And we can see that we're forbidden. You'd be surprised how many you'll find that will work without anything. But, you know, that's for another video. Now, I'm going to add back in the user agent. still forbidden because we know that we need the cookie. So, ah, still forbidden. And that's because there's one more in here that I know that my site is looking for and I think a lot of sites do this too and that's a good referer. They want to understand that this is actually come from somewhere. So, this refer referral header is quite important. But basically, what you want to do is go through this, figure out which headers you need, which ones you don't, and then start to look at the cookies. Now, another place you can see all the cookies is if we go back into Chrome and I go to application and we got storage and cookies here. We can see that this is the only one in this case. More actual real sites will have lots in here. So, you will need to start to figure out which ones you need and which ones you don't and you can build it out that way. Essentially, our goal is to get this as stripped back as possible to know exactly what cookies and headers we need and then understand how we can generate them and then understand how we can, you know, export this and run this as code. But I want to show you something else. So, at the moment, this works fine, but I'm going to go ahead and change my IP just through my VPN. And we can see we're forbidden, but we still have, you know, the good or token referer and the user agent. And that's because this uh OR token is linked to the IP address that has made the initial request. So if I disconnect this again, it works again. And what this means is that we really need to think about our fingerprinting and our session usage. So looking at this now, I know that we need to pass the JavaScript test, which is how we get the or token that's done with the browser. And we also need to link that to an IP address because it's being checked for every time we make a request. We'll also need to consider how long this orth token will survive for. Um, I can tell you right away because I built this, it's 5 minutes. So, you know, once we get to 5 minutes, this will expire and you need to regenerate it again. So, that leaves us in a little bit of an interesting place because we can't just get an off token or a good cookie, use that in our requests, and just, you know, rotate IPs. We don't want to send loads of requests through one IP because we know that that will just get us blocked very quickly and easily. It's very common way to do things. So, how do we do this? Well, one of the best ways is to build up a small storage unit that will have a good token and IP assigned to it. So, we will load up a browser with an IP address. We'll get the call token and get that sorted. We'll link that to the IP address and then store it. We can do that, let's say, three or four, five times. And then every time we make a request programmatically through our code, we can use one of these. Then we can build in a fail safe. So if we get a you know if if it fails the cookie expire the to the cookie and the token expire or the IP gets blocked we can then retry the request with a new one and keep topping it up. What this unfortunately means is that there is a lot more overhead required. You need to really think about how you're going to generate that cookie and that session ID to be able to consistently scrape. And we all know that browsers are quite heavy and a bit finicky. they like to work sometimes and not other times. So finding out what the best approach to that is going to be built into this. So overall what happens is your actual web scraping code becomes very minimal and the infrastructure that you need to be able to build out something that works reliably becomes much larger. And this is one of the biggest problems that a lot of people don't consider when they start to scrape data. So, for me, if I was going to build this out, and I think I will in a future video so I can show you, we're going to need some kind of proxy management system. We're going to need a browser management system. We're going to need a storage for the good cookies and and uh headers and um IP addresses so we can then use those in our requests. Our actual request, our scraping part is going to be one Python function. So, there's a lot to think about here. Now, obviously here at Zite, we do this all the time and we have our Zite API which would handle all of this and manage it all for you. So, you know, if you didn't want to do this yourself, you can just send our API the URL and it will manage everything and you'll just get the data straight back. So, there is that to consider. But that's going to wrap this video up. Make sure you subscribe. There's going to be more along these lines, more highlevel and more in-depth, sorry, stuff into more technical web scraping content. So, thank you very much and I will see you again in the next