ArticleBuild-throughAnti-ban

Inside the anti-block engine: How we built a 320,000-strategy web access hero

For data gatherers, negotiating anti-bot tech is not for the faint-hearted. That’s why we built a multi-layered system that treads lightly but brings a host of tactics to the table.

Akshay Philar

6 min read ·

Inside the anti-block engine: How we built a 320,000-strategy web access hero

A few years ago, we noticed a fundamental shift in how the web defends itself.

For a long time, web scraping followed a predictable cat-and-mouse dynamic. You built a spider, pointed it at a target, and wrote a parser. If you got blocked, you rotated your IP addresses, tweaked your user-agent headers, and went back to work. It was a manual, iterative game, but it was highly manageable.

Then came what I call the “Great Hardening”.

Target sites stopped relying on simple IP blacklists. Instead, they began deploying sophisticated, commercial anti-bot platforms. These platforms don’t just look at where a request is coming from; they analyze TCP/IP stack fingerprints, TLS handshakes, browser engine behaviors, and real-time user interaction patterns.

Suddenly, the support tickets we received weren't about simple mark-up changes. They were about silent, complex failures: sessions that mysteriously expired after three requests, browser fingerprints that burned out instantly, CAPTCHAs appearing mid-flow, and "soft-blocking" where a site returns a 200 OK status but delivers an empty page or a challenge screen.

And the rate of change accelerated. Where sites once made a periodic change every few months, AI-powered antibot platforms started pushing alterations faster than humans could stay ahead.

That was the moment we had to accept a brutal engineering reality: at scale, ban handling is no longer a configuration problem; it is an AI engineering problem.

To solve it, we had to stop building better bypass tools and start building an automated decision engine. Today, that engine orchestrates over 320,000 strategy permutations per request. This is the story of why we built it, how it works under the hood, and what we learned along the way.

The ‘best config’ fallacy

When engineering teams try to build an anti-ban system in-house, they almost always start by assembling a toolkit. They buy a few proxy pools, write some retry logic, integrate third-party CAPTCHA management, and write custom browser automation scripts for their toughest targets.

But the problem isn't the tools themselves; it's the orchestration. On every single request, someone - or something - has to make a series of high-stakes decisions:

  • Can we get away with a lightweight HTTP request, or do we need to pay the massive latency and CPU tax of spinning up a headless browser?
  • Which proxy network and geographic location will trigger the least friction for this specific target right now?
  • How do we manage sessions and cookies so we look like a returning user without creating an obvious, machine-like footprint?
  • If a request fails, what does the failure actually mean? Is it a transient network hiccup, an aggressive rate limit, or a hard IP ban?

If your team is making these decisions manually by writing custom configuration files for every target domain, you are playing a permanent, exhausting game of whack-a-mole. The moment an anti-bot vendor updates its detection algorithms, your static "best config" instantly decays.

We realized that the only way to win was to move the burden of adaptation from human engineers to an intelligent control layer. That engine's job is to choose the lightest possible strategy that works, and to constantly revise that choice based on real-time feedback.

The power of a strategy permutation

When we say our engine navigates 320,000+ strategy permutations, we aren't throwing around a vanity metric. That number represents the actual mathematical complexity of modern web access.

A "strategy" is not a single magic bypass trick. It is a highly coordinated bundle of choices across multiple technical dimensions, as shown in the table below:

Dimension Technical choice Impact on access
Fetch mode Raw HTTP vs. headless browser rendering Determines if we pay the CPU and latency cost of a full browser.
Identity profile User-agent, TLS fingerprint, TCP/IP stack Matches the client's network-level signals to look like a real consumer device.
Geo-coherence IP location matching DNS, CDN, and header locales Prevents blocks triggered by "impossible travel" or regional mismatches.
Session policy Session reuse, state management, cookie isolation Allows us to maintain logged-in or authorized states across requests.
Escalation rules Automated CAPTCHA management, interactive clicking Solves active friction points only when they are presented.
Retry strategy Adaptive backoff, error-specific retry budgets Prevents us from "hammering" a site and burning our own IP pools.

If you multiply these options across dozens of proxy networks, hundreds of geographic regions, and thousands of browser profiles, the search space becomes massive.

The real magic of the engine isn't that we have these 320,000 options; it's that the signal layer knows how to interpret failures so we don't guess blindly.

If a site returns a "soft block" (like a challenge screen wrapped in a 200 OK), the engine doesn't just retry the request with the same settings. It reads the signal, realizes the fingerprint is burned, escalates the request to a browser context to solve the challenge, captures the session cookie, and then drops back to cheap HTTP for the remaining requests.

This is what we mean by "the lightest strategy that works." It keeps success rates high while keeping our customers' API costs and latency as low as possible.

The reality of constant evolution

There is a common myth in the scraping industry that you can write a "perfect" unblocker, package it up, and sell it as a static product.

The reality of production web scraping is closer to Site Reliability Engineering (SRE). The web is a highly dynamic, living organism. Anti-bot vendors are constantly shipping updates, target sites are switching security providers overnight, and machine-learning models are continuously updating their detection heuristics.

This is why a solo developer or a small engineering team trying to build equivalent functionality in-house is fighting a losing battle. It isn't a question of engineering talent; it is a question of feedback loops and R&D bandwidth.

Because Zyte API processes billions of requests every week, our engine receives a massive, continuous stream of telemetry. We see anti-bot updates the moment they are deployed anywhere on the web. Our automated loops catch and resolve the vast majority of these changes before customers even notice.

And for the genuinely new defensive techniques - the ones that require creative human engineering - we have a dedicated, full-time team of anti-bot researchers. Their entire job is to study emerging security technologies, design new bypass mechanics, and feed those capabilities into the engine's library.

Building a system like this requires an ongoing, multi-million-dollar commitment to research and development. It is a continuous, self-healing loop of signal, decision, configuration, and human R&D.

The web isn't going to become easier to scrape. But, by moving the burden of adaptation from human engineers to an automated decision engine, we can at least make the fight sustainable.

We didn't build an anti-ban engine because it was an elegant engineering exercise; we built it because, at our scale, it was the only sane way to survive.

Try Zyte API

Build your first scraper in minutes

Free trial, no credit card. From a single request to production in an afternoon.

Get started

Akshay Philar

Akshay Philar is focuses on alignment across Platform Engineering, Data Delivery Engineering, Product Engineering, R&D and Tech Operations. He helps shape engineering strategy, strengthen the organization and scale Zyte’s engineering platform.

More from this author

Continue reading

Group on stage at Extract Summit Dublin 2026
Web data collection

Five takeaways from Extract Summit Dublin

AI is making scraping code cheaper to write. Extract Summit Dublin explored the harder work that remains: access, infrastructure, maintenance and trust.

Robert Andrews
A stream of web pages flowing from a globe through a gateway into a vector database
Data gathering for AI

Web Scraping for RAG: Build a Reliable Pipeline with Zyte API, Qdrant and LangChain

Build a RAG pipeline that survives bot blocks and JavaScript pages: fetch LLM-ready Markdown with Zyte API, gate bad pages, and test it before deploying.

Ayan Pahwa
Akshay Philar speaking at Extract Summit about the anti-ban arms race and the open web.
Anti-ban

The anti-ban arms race: Agentic traffic and the fight for the open web

As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.

Akshay Philar9 min read
One scrapy-playwright Spider, Three Browser Setups: Playwright, Patchright, and Zyte
How To

One scrapy-playwright Spider, Three Browser Setups: Playwright, Patchright, and Zyte

One Scrapy spider, three browser setups: stock Playwright, Patchright via the new PLAYWRIGHT_BROWSER_PROVIDER hook, and a remote browser on Zyte over CDP. Same spider and selectors throughout, only the browser changes.

John Rooney
K1
Anti-ban

The recipe for a request: Scaling data extraction through investigation

Learn how an investigative mindset helps scale data extraction from single requests to millions daily by building resilient, efficient scraping systems.

Kieron Spearing5 min read
Hybrid scraping: The architecture for the modern web
Anti-ban

Hybrid scraping: The architecture for the modern web

Learn how hybrid scraping combines headless browsers and lightweight HTTP clients to bypass JavaScript challenges efficiently. Reduce RAM usage, improve speed, and scale your web scraping pipelines with session reuse and TLS fingerprinting.

John Rooney10 min read

The Community · Newsletter

The best of Zyte and the data web, in your inbox.

One curated edition — new articles, product updates, and the stories shaping the data web. No noise.