Your Web Scrapers Keep Getting Blocked. Here's Why

Anti-botting is no joke. It's been steadily getting better and better year on year with many websites that used to be easier to scrape adopting new technologies in an attempt to stop their public data being extracted. And with the rise of AI scraping bots, many vendors are just outright blocking by default, no intervention required by the website owner. In this video, I want to share some of the advice on how to manage these bans coming from a webinar I did with two of Zyte's top anti-ban team members who've been working in this space for decades. I want to share with you some actionable tips and talk a little bit about how we manage it here. So, there are two key points we need to dive into. The first is the initial steps you need to take when working out how you're going to scrape a site. And the second is how to implement a better crawl strategy to make sure you go under the radar. It's a mix of strategy plus the tooling and infrastructure that you use that will enable you to scrape successfully. To understand what we need to do when working on a site, we must first understand the measures that can be taken in a modern multi-layer anti-bot solution to block our access or give you a challenge. Put yourself in the shoes of an anti-bot engineer. How would you try and separate real traffic from bot traffic? Now, this can be broken down into two areas: fingerprinting and session usage. Your fingerprint is a unique identifier to your specific user as the website sees it. A lot of information can be gathered by running JavaScript on your browser and likewise, if you aren't running a browser at all. There's a lot to consider here given the sheer number of data points that can be gathered from your machine through this method. More obvious ones like user agent and operating system, right the way down to your laptop's battery life, what fonts you have installed, your video card, and even your mouse movements. Yeah, that's right. Mouse movements can play a part here. This is important as often when automating a browser, the mouse jumps between points in an impossible manner for a human to do. And this data can then be used to serve a challenge, effectively blocking your access. All this data is then compiled into a fingerprint to identify you, usually in a hash. It's this point here that often stops and confuses newcomers, as we're led to believe that rotating proxies is a way around bans. Whilst that is true, it's just not the full picture, and this fingerprint can be used as well to rate limit or challenge regardless of what IP you're using. Whilst we're talking about IPs, it's important to think of an IP as something that has a quality score associated with it. Rather than a just a location and its type, although these do play a role, it's the quality that will give you the access. We use a mixture of different proxy IPs through our Z API to gain access and reduce the cost. So, why is this fingerprint so important? Well, let's think about a common example, scraping data using Selenium or Playwright. Without any modification, both of these browser automation systems have a very unique and distinctive fingerprint, allowing instant blocking if any requests are made with them. They can be patched, for sure, but this is just a massive cat and mouse game, trying to stay one step ahead. Another example, I mentioned fonts earlier. If your machine is identified as macOS, for example, but the fonts show Times New Roman installed, which isn't there by default, your reputation drops. Could someone have legitimately installed it? Of course, but still, it's a contributing factor. One thing that caught me out a lot was the difference between time zones of your IP and your browser. This is especially true when rotating through different proxies globally within the same browser locale. To recap, our first point is to maintain a consistent fingerprint as a real user would. Be mindful of what data is exposed and how you choose to manage it. Be consistent and remember that the goal is to create a session within the website that looks as if a real user was operating it. But how do we know what we need to do? Well, this leads into the second point, understanding what antibot system the site has, how it reacts to certain triggers, and building a clear picture of what your crawl strategy should look like. As an example, at common antibots use a session cookie stored locally. Once your initial checks have been passed, you get this cookie assigned to you. And now this is your ticket that you need to manage. In some cases, this is all that's required. But in others, changing something like the IP could cause a challenge as this could be considered unusual user behavior. But for some, once this session is obtained and the cookie is working, you can just drop the browser all together, change the IP to achieve real alternatives, start making requests without a browser, all that stuff. But you will need to monitor the site and understand what changes you make and how it responds, and then take action accordingly. Also, take into consideration how a real user would browse a site. It's less likely they'd go straight to a page deep within the site, albeit not impossible, but it's very unlikely that they go page by page, product by product, all in order. Your goal is to keep it natural whilst maintaining a good session with constant fingerprint and crawl the site in the correct manner. Even going to the home page first before you design pages can make a difference in some cases. But there's one more thing I want to mention that I found interesting from the webinar, and that's AI usage. Antibot vendors have been using machine learning for years to spot patterns with fingerprinting and behavior, but with AI now it's even easier and even better at spotting anomalies like the ones we talked about earlier. But it's also useful the other way around, helping to deobfuscate the huge amount of JavaScript on the web that isn't meant to be read easily. I know it's more low-level knowledge, but you know, useful things to know. But this is all a lot. The reality is that staying ahead of the modern antibot tech is huge amount of work, and depending on the data required could mean it's out of reach for a lot of businesses, and that's where we can help. The Zyte API handles everything I just mentioned. No need to worry about IP quality, sessions, or fingerprint. Just send a simple request to our API and receive the raw HTML back. We are very good at this. So, find the link below, sign up for free credit, and let us show you.