How to transfer browser cookies to an http session when web scraping

I want to show you today a web scraping technique that I've been referring to as hybrid scraping. It's a bit like session management, but it's a way of using a browser to generate cookies and or headers and then sending them along with further HTTP requests from a client HTTP client, which is much more cost- effective and faster and lighter on memory. I'm going to explain to you in this video when you should be using this, the benefits of it, the downsides, and also explain how we use this in our product, the Z API, and I'll show you an example using all opensource too. So, what I've got here is a very very basic life cycle representation and I want to explain to you how this is going to work. So up here is our request and if we were to make a request to the website down here following through the browser route like so we would get through the fingerprint check which is in the antibbot layer get the cookie generated and then have access to the data. This is fairly simplified but kind of understand the process here. But this part here is very very expensive. It's very very memory heavy. It's complicated. is infrastructure costs and also extra data costs depending on how you're actually rendering those requests. If we were to make the same request through an HTTP client like requests or similar, we might find that we would get to this part here. We would fail the fingerprint check because we don't have the ability to run that JavaScript test which is going to be giving us that cookie. So, we end up over here in the 403 land. So what we want to do with this is we want to trace this request down here, generate this cookie and then take it back up here and in and inject it into our session. So we can make requests like this whilst our cookie is still good and active. Then as it gets more complicated, we can actually do this here generate multiple cookies and use them through here. Now this is kind of like a precursor as I me mentioned before as um as to working with sessions. A session is like involves the cookie, all the headers, the sort of the fingerprinting as well from the browser as well as the IP. That's a more complicated version, but you'd be surprised how far you can get doing it just this simple way. So, here's the code example that I've got open on my screen here. Let's just make this a bit bigger so we can see. Now, this is a test website that I have um I have developed and I know how this works. I put in some antibbot technology into it. So if I open this up here, you can see we get this checking browser thing and then we can access the the data on the pages. If I go to inspect on this, bring this out over here and go to application and we can see local storage under storage just session storage cookies. There we go. We can see that we have this cookie here. Uh you can probably just about see this over in the corner. Uh I conveniently called it O token here but you'll quite commonly see it related to whatever antibbot technology or vendor is being used on that website. They have their own cookie names. Now without this cookie I'm not able to access this website without passing the JavaScript test. So for example if I delete this cookie from my storage and refresh we're going to get the checking browser thing again and we get it back. Now in this instance the browser actually gets passed through the checks and the uh if you was to do this through a simple HTTP request it would get blocked because we aren't able to pass that JavaScript test to generate this cookie. So what we can do is we can work like this. So I'm using here playright which is a very common headless browser or headful browser depending on how you want to do it. I'm also using ccfi with requests. So, this is just a modified version of the request package which allows us to send a more TLS friendly fingerprint to websites. I'm not going to cover that in this video, but I'll cover that in subsequent ones and also some blog posts. So, there'll be links to that down below if you want to find out a bit more about TLS fingerprinting. All I'm doing here is creating a session that is going to look like Firefox. However, this on its own without the cookie isn't going to get us through. But this is where we're going to load up our browser. We can see that I'm just creating a new browser instance here. And then I'm basically adding in this initialization script. So when the page is opened, it removes the web driver property. Now for those of you that aren't aware, this is the most common giveaway for headless browsers like Playright and etc. Um that they have this property in their flag that is basically says web driver is equal to true and that is a very clear giveaway. This is actually a very old way of detecting and um it's very easy to remove. So, it's not really relied on anymore as a um method of getting past any protection on the website. However, I put it onto my test site just as a demonstration. Then we're going to go ahead and grab the cookies from the page and I'm going to take them and I'm going to load them into my session which is going to allow me then to create and use the session to make requests to this page on the website which is obviously a different page but we'll see them working. So what I'm going to do first is I'm just going to comment out the uh adding the cookies to the page and we're going to run this when I get my terminal back up. UV run main.py py. This is going to pop up the uh check browser and we can see that it's got through here. And then we have our challenge here again down here. We can see somewhere under here. Yeah, I can't remember what I called this. Did I give this page a title? Yeah. So, I called it security check. This would be, you know, the most common one you'll see is just a moment or something similar to that. Please wait while we verify. This is because without these cookies, I don't have access to the website. So, what I'm going to do is I'm just going to put these back in. We're going to take these cookies and I'm going to clear this up and we'll do UV run again. Passing the browser check with the T with the browser generating the cookie injecting it into our HTTP request session. And then here we are actually have access to that page. Now, there's no data on this page. I use this website uh for multiple different tests. So, you know, you can see here that this page data is actually being loaded by JavaScript. The important thing here is that we got through the challenge and we would be able to do so for as long as that cookie is valid. This is actually a very kind of like a precursor to as I said sessions but also using a browser as little as possible but you do actually need it. So in the ZI API, we have this uh over here. And if I show you um under this section here, make this a little bit bigger. You can actually set your own cookies when you're making requests uh through our uh REST API to to scrape data or you can pull the cookies out here. And this is this is our example of what I just showed you through the open source programs the uh the open source packages is that here we're basically saying you know here's the URL give me the response cookies and then we're passing them into a non-browser request. So you can see browser HTML in the first request and then we don't we have HTTP response in the second request. So this is basically doing the same thing but we're handling all the infrastructure for you in that instance. Now, behind the scenes, we do a lot of this anyway because, you know, this is a more efficient way to scrape. So, if you're in that place where you don't want to manage this sort of thing yourself and you just want the data back, then you can absolutely just send us the request and we will handle it for you. But this is well worth knowing how to do. So, just to just to clear that up one more time here, we are loading the browser app once, generating the session, the cookies, the headers, whatever else you need. You might need multiple different cookies. In this case, my test site was only one. And we're loading them all into our actual HTTP session for our HTTP client. And then we can utilize that with sub subsequent requests. So hopefully this has helped you out in some way. Make sure you leave me a comment, like down, like the video as well. Always helps. There'll be more of this stuff to come. And I'll leave some links in the description for written versions of this that you might find beneficial.