ArticleAnti-ban

Use Powerful Ban Management With Zyte API’s Scripting API

You encounter so many different anti-bot systems as a developer working on custom projects for Zyte Data clients. This cat and mouse game requires your time, persistence and deep knowledge of web scraping.

Mushtaq Ali

5 min read ·

Use Powerful Ban Management With Zyte API’s Scripting API

You encounter so many different anti-bot systems as a developer working on custom projects for Zyte Data clients. This cat and mouse game requires your time, persistence and deep knowledge of web scraping. We have a story to tell you about one of these games. It was a daily battle that was slowing the team down. In this post, we’ll break down the problem and show you how we solved it with Page.evaluate() from Zyte API’s Scripting API.

Our client has high demands. We need to deliver data according to three different schedules: hourly, daily, and fortnightly. There isn’t much room for error when you have hourly and daily schedules. Access to websites is of paramount importance to ensure the client gets the web data that’s absolutely essential to their operations.

The headache of managing daily bans

We migrated the client's spiders from Smart Proxy Manager (SPM) to Zyte API with cookies and headless browsing. We had three glorious months free of anti-bot issues before they slowly reintroduced themselves. 

It started with occasional bans and increased to weekly bans. We started using the Scripting API endpoint and a few header changes to overcome these early bans. However, the bans started to happen more frequently and our team would respond with manual fixes each time. We guessed that the website was identifying our crawler by analyzing the headers. So, we would spend hours cycling through different combinations of header and cookie logic to find the one that allowed website access. That fix would last a couple days, if we were lucky, and the cycle would begin again.

This was urgent client work, so the team was working around the clock; the bans happened at all times of the day and night. The manual fixes were difficult. Finding the configuration that allowed website access on that particular day was time consuming and frustrating. Want to know what demoralizes your web scraping development team? Never ending ban situations like these.

The development team’s velocity slowed right down. We were spending so much time figuring out the configuration of the day. It was critical to find a solution that would stick.

Scripting API to the rescue

We contemplated writing a solution that would save us time by automatically discovering the winning combination. At the beginning of each day, it would cycle through all the different combinations of headers until it found that special one. The spiders would then be set to use that combination. This wouldn’t really solve the fundamental problem of our crawlers being identified, but it would help the development team with the gruntwork. 

We discovered that using browser scripting to replicate a user session was the trigger that tipped off the anti-bot system. We needed the browser to access the website but needed to find another way to execute user actions so they’d be undetected. 

Zyte API’s Scripting API has a function called Page.evaluate(). It allows us to execute JavaScript (JS) code within a page context. So instead of using the browser scripting to click different elements in the user session, we used JS to program the actions and called Page.evaluate() to execute that code. Once we had access to the site, all the requests came from the same browser and IP, making it hard to be detected, and the JS performed the user actions faster. 

Our Scripting API strategy eliminated the excessive amount of time used to setup and discover the winning combination of headers. JS was the perfect programming language to perform a request and process a response from the site as if you were using a browser. Our team’s development velocity (and sanity levels) returned to normal because we weren’t spending hours battling website bans. Now the client has consistent access to the target domain.

Interested in trying out Zyte API’s powerful ban handling features like the Scripting API? Sign up for your free trial and see how we handle your toughest websites.

FAQs

What is the challenge Zyte API's Scripting API solves?

It simplifies handling anti-bot systems by automating actions with JavaScript and reducing the need for manual header configurations.

How does Page.evaluate() help with web scraping bans?

It executes JavaScript within a page to replicate user actions without triggering anti-bot systems, making sessions harder to detect.

What benefits does the Scripting API offer developers?

It eliminates time-consuming manual fixes for bans and restores development speed by automating header and cookie management.

Why is JavaScript used for ban handling in Zyte API?

JavaScript mimics browser-like behavior efficiently, allowing seamless user actions while avoiding detection by anti-bot systems.

How can I try out Zyte API’s Scripting API for my projects?

Sign up for a free trial to test its powerful ban handling and automation features on your toughest web scraping challenges.

Try Zyte API

Build your first scraper in minutes

Free trial, no credit card. From a single request to production in an afternoon.

Get started

Continue reading

Illustration of an e-commerce storefront secured inside a vault
Access handling

The world's hardest sites protect their prices

Only a sliver of the web demands the heaviest machinery to read. It is dominated by shops, property listings and booking engines, where every published price is somebody's market intelligence.

Robert Andrews
Rows of golden keys converge on the keyhole of a dark door, illustrating multiple strategies for gaining access.
Anti-ban

Inside the anti-block engine: How we built a 320,000-strategy web access hero

For data gatherers, negotiating anti-bot tech is not for the faint-hearted. That’s why we built a multi-layered system that treads lightly but brings a host of tactics to the table.

Akshay Philar6 min read
Group on stage at Extract Summit Dublin 2026
Web data collection

Five takeaways from Extract Summit Dublin

AI is making scraping code cheaper to write. Extract Summit Dublin explored the harder work that remains: access, infrastructure, maintenance and trust.

Robert Andrews
A stream of web pages flowing from a globe through a gateway into a vector database
Data gathering for AI

Web Scraping for RAG: Build a Reliable Pipeline with Zyte API, Qdrant and LangChain

Build a RAG pipeline that survives bot blocks and JavaScript pages: fetch LLM-ready Markdown with Zyte API, gate bad pages, and test it before deploying.

Ayan Pahwa
Akshay Philar speaking at Extract Summit about the anti-ban arms race and the open web.
Anti-ban

The anti-ban arms race: Agentic traffic and the fight for the open web

As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.

Akshay Philar9 min read
One scrapy-playwright Spider, Three Browser Setups: Playwright, Patchright, and Zyte
How To

One scrapy-playwright Spider, Three Browser Setups: Playwright, Patchright, and Zyte

One Scrapy spider, three browser setups: stock Playwright, Patchright via the new PLAYWRIGHT_BROWSER_PROVIDER hook, and a remote browser on Zyte over CDP. Same spider and selectors throughout, only the browser changes.

John Rooney

The Community · Newsletter

The best of Zyte and the data web, in your inbox.

One curated edition — new articles, product updates, and the stories shaping the data web. No noise.