A few years ago, we noticed a fundamental shift in how the web defends itself.
For a long time, web scraping followed a predictable cat-and-mouse dynamic. You built a spider, pointed it at a target, and wrote a parser. If you got blocked, you rotated your IP addresses, tweaked your user-agent headers, and went back to work. It was a manual, iterative game, but it was highly manageable.
Then came what I call the “Great Hardening”.
Target sites stopped relying on simple IP blacklists. Instead, they began deploying sophisticated, commercial anti-bot platforms. These platforms don’t just look at where a request is coming from; they analyze TCP/IP stack fingerprints, TLS handshakes, browser engine behaviors, and real-time user interaction patterns.
Suddenly, the support tickets we received weren't about simple mark-up changes. They were about silent, complex failures: sessions that mysteriously expired after three requests, browser fingerprints that burned out instantly, CAPTCHAs appearing mid-flow, and "soft-blocking" where a site returns a 200 OK status but delivers an empty page or a challenge screen.
And the rate of change accelerated. Where sites once made a periodic change every few months, AI-powered antibot platforms started pushing alterations faster than humans could stay ahead.
That was the moment we had to accept a brutal engineering reality: at scale, ban handling is no longer a configuration problem; it is an AI engineering problem.
To solve it, we had to stop building better bypass tools and start building an automated decision engine. Today, that engine orchestrates over 320,000 strategy permutations per request. This is the story of why we built it, how it works under the hood, and what we learned along the way.
The ‘best config’ fallacy
When engineering teams try to build an anti-ban system in-house, they almost always start by assembling a toolkit. They buy a few proxy pools, write some retry logic, integrate third-party CAPTCHA management, and write custom browser automation scripts for their toughest targets.
But the problem isn't the tools themselves; it's the orchestration. On every single request, someone - or something - has to make a series of high-stakes decisions:
- Can we get away with a lightweight HTTP request, or do we need to pay the massive latency and CPU tax of spinning up a headless browser?
- Which proxy network and geographic location will trigger the least friction for this specific target right now?
- How do we manage sessions and cookies so we look like a returning user without creating an obvious, machine-like footprint?
- If a request fails, what does the failure actually mean? Is it a transient network hiccup, an aggressive rate limit, or a hard IP ban?
If your team is making these decisions manually by writing custom configuration files for every target domain, you are playing a permanent, exhausting game of whack-a-mole. The moment an anti-bot vendor updates its detection algorithms, your static "best config" instantly decays.
We realized that the only way to win was to move the burden of adaptation from human engineers to an intelligent control layer. That engine's job is to choose the lightest possible strategy that works, and to constantly revise that choice based on real-time feedback.
The power of a strategy permutation
When we say our engine navigates 320,000+ strategy permutations, we aren't throwing around a vanity metric. That number represents the actual mathematical complexity of modern web access.
A "strategy" is not a single magic bypass trick. It is a highly coordinated bundle of choices across multiple technical dimensions, as shown in the table below:
| Dimension | Technical choice | Impact on access |
|---|---|---|
| Fetch mode | Raw HTTP vs. headless browser rendering | Determines if we pay the CPU and latency cost of a full browser. |
| Identity profile | User-agent, TLS fingerprint, TCP/IP stack | Matches the client's network-level signals to look like a real consumer device. |
| Geo-coherence | IP location matching DNS, CDN, and header locales | Prevents blocks triggered by "impossible travel" or regional mismatches. |
| Session policy | Session reuse, state management, cookie isolation | Allows us to maintain logged-in or authorized states across requests. |
| Escalation rules | Automated CAPTCHA management, interactive clicking | Solves active friction points only when they are presented. |
| Retry strategy | Adaptive backoff, error-specific retry budgets | Prevents us from "hammering" a site and burning our own IP pools. |
If you multiply these options across dozens of proxy networks, hundreds of geographic regions, and thousands of browser profiles, the search space becomes massive.
The real magic of the engine isn't that we have these 320,000 options; it's that the signal layer knows how to interpret failures so we don't guess blindly.
If a site returns a "soft block" (like a challenge screen wrapped in a 200 OK), the engine doesn't just retry the request with the same settings. It reads the signal, realizes the fingerprint is burned, escalates the request to a browser context to solve the challenge, captures the session cookie, and then drops back to cheap HTTP for the remaining requests.
This is what we mean by "the lightest strategy that works." It keeps success rates high while keeping our customers' API costs and latency as low as possible.
The reality of constant evolution
There is a common myth in the scraping industry that you can write a "perfect" unblocker, package it up, and sell it as a static product.
The reality of production web scraping is closer to Site Reliability Engineering (SRE). The web is a highly dynamic, living organism. Anti-bot vendors are constantly shipping updates, target sites are switching security providers overnight, and machine-learning models are continuously updating their detection heuristics.
This is why a solo developer or a small engineering team trying to build equivalent functionality in-house is fighting a losing battle. It isn't a question of engineering talent; it is a question of feedback loops and R&D bandwidth.
Because Zyte API processes billions of requests every week, our engine receives a massive, continuous stream of telemetry. We see anti-bot updates the moment they are deployed anywhere on the web. Our automated loops catch and resolve the vast majority of these changes before customers even notice.
And for the genuinely new defensive techniques - the ones that require creative human engineering - we have a dedicated, full-time team of anti-bot researchers. Their entire job is to study emerging security technologies, design new bypass mechanics, and feed those capabilities into the engine's library.
Building a system like this requires an ongoing, multi-million-dollar commitment to research and development. It is a continuous, self-healing loop of signal, decision, configuration, and human R&D.
The web isn't going to become easier to scrape. But, by moving the burden of adaptation from human engineers to an automated decision engine, we can at least make the fight sustainable.
We didn't build an anti-ban engine because it was an elegant engineering exercise; we built it because, at our scale, it was the only sane way to survive.







