Zyte API Sessions - flexible cookie management maintaining control

In my last video, I talked about how to use a hybrid scraping method where you load a browser up and then take the cookies and the headers and the information and then you can transfer that across to an HTTP client session. And in this video, I want to expand on that and show you how we do it within Zite API and how it's much much easier to manage and handle. So this was the demo site and if I go to inspect and just open this up and if we look at the application we can see that we have this or token cookie. Obviously this is my demo site. Real sites will look slightly different but if I delete this and refresh the page we get this browser check. What this means is that if we aren't able to pass the JavaScript test that runs on this browser when you initially load up the site that generates this token, you're going to get rejected and you're going to be blocked or whatever is going to happen. So what we want to do is load this up. Now within site API, we have the browser feature here which will you know use a browser through our system which will render the HTML and work for you. Now, you could of course use this to scrape every page, but this is, you know, slower, it's heavier, it's uh it's more costly because we're running more on the infrastructure. But, as I talked about at the beginning of the video, we can actually use one browser request and then utilize that session to then make multiple requests. And what's different about this site is that we don't just want to make multiple requests with the HTTP to the page because if I go view page source, you'll see that this page is loaded dynamically. So we need to go one step further and we need to actually utilize the network page that is being the network request sorry that is being uh run here and will actually then return the JSON data. Very standard way of scraping websites like this. So what do we need to do then? Well, we can have a look at the features. And if we look at uh sessions, which is here, we have two options, client managed and server managed. I'm going to be going through client managed in this one because I want full control. So if we scroll down to the example and I look under the examples, this is the one that I want here. Reuse browser cookie session in reuse browser cookies and HTTP requests. It's exactly what we want to do. From here we can see that we are making two requests with a session ID which is a UYU ID for nice and straightforward in Python. So what I did is I copied this out and I recreated it here and I just need to edit this one. There we go. So I recreated it here. Now let's make this a bit bigger. So I'm pulling in my Zite API key from my environment here. And then we're going to go ahead and create our browser request just like we did manually except you this time we don't have to worry about running the browser or managing it or anything like that with the session ID. And with this session ID we can then reuse it. Now what I've got down here is the actual API response page that I needed to get to to actually get this information back. We saw here this was the URL that gives me this information which in this case is the product information. Now what happens is we send in our session ID which we created which we've utilized and this is going to reuse it. So if I go ahead and run this and I've got that commented out for a reason. I just want to show you this this response. So Zite API is running the browser for us. It's then taking the cookies and the session from that transferring it over to the HTTP requests and then going to run that one. So, the first one is always going to take longer, but once you actually have that loaded up and have that session, you'll be able to use it multiple times. Now, I'm getting a key error here, and that's because we have um this here. So, what I want to do is I'm going to show you that in the cases where you're trying to hit an API endpoint with this, you might come across this error. So, I'm just going to do print uh HTTP response.json, and we'll run it again. What we're going to see is that we have an issue with some of the headers. Now, in this web on this website, obviously I made it. I know which headers you need to include to actually get that request. But our API is going to respond and say, you know, website ban. You think, oh well, it didn't work. But actually, what's going on here is it's saying, if you are targeting an API endpoint, please see this documentation. And that's exactly what we are doing. So I'm going to come over here and we'll load this in the page and it talks about down here when targeting API pens set the right request headers and cookies. So to me straight away I know that my headers are probably not correct. And if you've done your due diligence and you've looked through this website or whatever website it is that you're trying to work and you've tested it with different curl commands or maybe an API um testing program, you'll know that there is a header here that we need to include to actually make this work. So I'm going to go over here and I'm going to put that header back in. And we put it under custom HTTP headers. And we can uncomment this and comment this out. Now, so we needed this referral. Now, our API documentation will talk about referers and say, you know, probably don't set them, but generally that's probably the case. I just made this test site that required these extra headers and I just picked that one. So, you just have to understand what you're working with. So, let's run this now. and we're going to send our request through the uh Zite browser. It's going to take the the the cookies and all the everything else. It's going to send it through the HTTP request and we're actually going to get this back. Now, there is another way that you can get this information because this is actually a network request. You can do this through the ZI API browser. There is the uh where is it? Network capture. But what this is for is this is going to capture all the network requests when you load that page up. it's not going to allow us to utilize those cookies to push them into HTTP requests. So, we actually wanted a different feature. It's just worth mentioning because you can get a bit confused if you look at this. You might think it's what you're after. So, from here, we would just want to tidy this up to actually, you know, loop through all the pages. Maybe we would put uh this into a function where we create our session and then we can like, you know, go through and send the different URL with the different pages as we go through. But essentially what we've done here is we have taken our principle our idea of utilizing a browser to load up the main page generate or pass the JavaScript test generate the the tokens or the cookies or the whatever we need and we're storing that into a session on API which we can then reuse and actually make requests to the sites API to get that data. uh and we can do it much quicker, much easier, and much more cost-effective than we than and much more reliably than if we were to try and do it ourselves. Generally, trying to manage browsers is a bit of a pain. And keeping updated and making sure that that browser that you're using is sufficient enough to actually beat that [clears throat] challenge is always going to be difficult to stay on top of. And this is a nice neat cost-effective solution.