3 Rules of Modern Web Scraping

In this video, I've got three tips for you that will improve your web scraping, improve the success rate of your request, and make your life passing HTML and getting data much easier. And the best thing about it is that they are all open source and free and very easy to implement. So, the first one is changing out your HTTP reclient. Now, for me, request has been a go-to for a very, very long time, and it's still very, very good. But for web scraping, it just doesn't cut it anymore. And it's because of its TLS fingerprint. TLS fingerprinting is a way that a client that a server can look at the client hello and figure out where the request is coming from and sort of what is sending the request based on some of the things like the cipher suites and some of the encryption methods. So this can be all turned into what's called a JA3 hash and matched against a library. And do you know what requests has a very very obvious signature? Let's look at it here. So I'm making a request using Python requests to this website here which gives us back our TLS fingerprint information. Here is the cipher suite available and here is the JA3 information. Now noticeably these are very very specific to requests and Python. This is a very very clear giveaway which combines into our JA3 fingerprint. So if I copy this out the fingerprint hash and we come and put it into this website here we can see right away that it's telling us hey we know from this hash that this has come from Python 3 and it's making a guess you know the library is URL3 which is what request is based on this is a very very clear telltale sign. It is not difficult for servers to block any request that comes from Python 3 like this. In fact it's very very easy and it's very common. We can see here that there's very there's loads of results. So this is not a good way. This will be regardless of the headers, regardless of cookies, regardless of your IPs, your proxies, you're still going to get stopped at the door if you use this method. So what can we do? Well, we can use a different package like curl cfi or in this case I'm using arnet. I'm going to do exactly the same thing here. So let's just run this. We should get this information back here. Now, we can see noticeably straight away that the cipher suites are different and specifically this one here, the TLS Grease, which is Google's one, which I think although it's not actually technically an encryption method, it's a way of just adding in some of the extra bytes of data. I don't know the ins and outs of what it exactly does, but I know that it's only there on a browser, specifically Chrome browser. So, this is a real big contributor to that JA3 hash. So if I come down here and grab it now, we can see that there's nothing found for this one. But if I come to the peak print hash, so we can kind of get an idea of what's going on, it's going to know that this is a browser and it thinks, you know, thinks it's Brave or Chromium kind of expected. So this just gives us that extra ability to not fall into that TLS fingerprinting issue just by changing out the HTTP client that we're using. In this case, as I said, arnet. Another good one is curl cfi. and they are based on two different things but very very good at what they're achieving. So the second thing that I want to show you is that when you're actually trying to work out how to process the data you want to be first look at the website and understand where the data comes from. I've got one example here but I'm going to talk you through another one. So when you start to flick through the pages on this website when I go back to the main page you start to flick through you can see that we have this exhr request coming up here. Now what this is is this is the front end of the website making a request to its backend API to get the data. So this is a very very common method very common way of splitting up a website. It will uh the front end will request the API to get that back. Now the resulting information is JSON data and we can actually just go ahead and mimic this request. We can do this ourselves through our uh Python code and actually just get this data directly rather than trying to you know like pass this information out. You can see this is exactly what's on here. Now this is a slightly simplified example but you'll be surprised if you just take the time how often you will find this on the website that you're trying to extract data from and make your life much much easier. I've got a video where I've gone into this in a bit more detail which I'll link below. But just bear in mind things like headers, cookies, referers, and user agents and your IP when you're trying to do this, but this is the way forward if you can do it. The second way is to actually look in the source code of the page. Now, we're not going to see anything here, but on sites like uh that are run by uh JavaScript frameworks like I believe next and next that are server side rendered within the actual source code of the page which is exactly what your Python script will see which isn't the DOM, two separate things. Then there will be a script tag often it uh with the type LD plus JSON with a very very strict schema that has all of the information in it in JSON format. It's very very similar to what we have here. It's just instead of you know this request being made at the time it's made when the page loads up and it's embedded into the page itself. So we can just make a request to the page but instead of having to worry about elements and you know HTML we can just find that script tag that's got that information in. So the first thing that I would say you know when you go page source is search for something that's on the page. For example of this one we could search for the name or I like to search for the word schema or you can even just search for LD plus JSON. Now if you come across this and you want to find an easy way to extract it, there is a Python package called extract which we have built and maintained free open source that manages it all for you. Very very easy and very very cool. So you can avoid passing any HTML. My third tip is that if you need to use a browser in any way, shape, or form, which is a little bit more common than you think, unfortunately, because it's a pain to actually manage, then you want to look at one of the more modern um stealthier browsers and stay away from all of the older stuff. So, the way that it kind of works is that once something has been out for a while, then the antibbot vendors know about it. they can patch the specific ways that it works to actually to actually bypass any of their defenses in a way. So what happens is we need to stay on top of it and we need to leverage the newest tools that come out and thankfully there's some really cool open source projects at the moment which are quite reliable. My favorite being no driver and it's for Zen driver and also camera fox. um they're both very popular and growing in popularity which is a very good thing but also you know a bit of a negative thing because again it's all about this fingerprinting that we need to worry about. But if I come and show you here I have this uh this is a standard playwright script which people a lot of people will use and try and leverage playright to render JavaScript or you know get into pages that they can't do with requests. And what happens is this. So if we run this script, we're going to see access denied and it was checking our browser. What's happening here is that this site is checking for some telltale signs that's that playright gives away and then finding them and then giving us a 43 and blocking us, i.e. not giving us that entrance cookie. But what we can do is use one of these other browsers that I mentioned just a minute ago. This one is Zen Driver. It's one of my favorites at the moment. And when we run this, we get access to the site. And you'll notice here that this started up actual Chrome, whereas the other one started up Chromium. Uh, and there's a few different ways that these work. So, I would highly recommend that if you do need a browser that you look into using one of the modern web scraping browsers. And then you also try to build out a browser session. So you don't have to load up loads and loads of pages with the browser because it will eat all your RAM and your memory and it will take you way more time. General rule and again I've got a video on this which I'll link. General rule is pass the JavaScript test with your browser. Save all the headers and the cookies that you need. Make subsequent requests with your HTTP client until you need to start the process again. So I'll leave a link to that video down below. Now of course these are all three things that I think are very very easy to change. drop in replacements in some cases and a couple of just different ways of thinking about things to give you a better chance at actually scraping that public data that you need. Now, of course, if you're still struggling with this and you need some help, then we can obviously be of assistance, check out the Zite API, which will handle everything that I've just talked about for you and just take in a URL and return you the data in a nice and simple, straightforward way. So, thank you very much for watching, guys, and I will see you in the next one.