I have a 60-year-old paperback on my shelf that belonged to my grandfather. It is one of the Sudden westerns by Oliver Strange, set in 1800s Texas. Growing up on the other side of the world, those books gave me a picture of Texas as an endless open range where cowboys rode across unfenced land as far as the eye could see.
The real history of Texas cattle country took a sharp turn in the late 1800s:
- 1874 - The Wire: A farmer in Illinois patented barbed wire, giving landowners a cheap way to fence off the plains.
- 1883 - The War: Cowboys who needed open land to move cattle began cutting the wire, sparking the Texas Fence Cutting Wars.
- 1884 - The Law: The state legislature made fence-cutting a felony, ending the open range. However, lawmakers added a vital rule: if you built a fence across a public road, you had to put a gate every three miles and keep it in repair for anyone to use.
That same story is playing out on the web today.
Over my nine years at Zyte building Zyte API, I have watched the open web get fenced off layer by layer.
Now, as AI agents change who uses the internet, our industry has to make sure the gates on those public roads stay open.
The 10-year arms race: From simple proxies to credit scores
In 2016, getting data from a website was straightforward. A datacenter IP address, a User-Agent header, and a basic JavaScript helper like py_mini_racer or jsdom handled most jobs. When Headless Chrome arrived in 2017 to replace tools like PhantomJS, developers building bot detectors and developers scraping data even shared their tricks publicly on Paul Irish’s headless-cat-n-mouse GitHub repository.
By 2021, that public sharing stopped. The turning point actually began in December 2016, when a major CDN bought a browser-fingerprinting company and deployed its bot detection across servers worldwide.
Today, when you request a web page, anti-bot systems inspect four layers before and during page load:
| Detection layer | What the website checks | Why it is hard to fake |
|---|---|---|
| 1. IP address | Where the connection comes from: a cloud server (datacenter), a static home/office line (ISP), or a real person's device (residential). | Datacenter IPs are cheap and flagged on sight. Passing checks often requires residential connections routed via app SDKs, ISP GRE tunnels, or physical device farms. |
| 2. Network handshake | The initial connection packets (TCP and TLS) before any HTTP data is sent. | Tools like p0f read packet settings to spot your operating system. In TLS, JA3 and JA4 fingerprint the browser's security settings even if you randomize their order. |
| 3. Browser control | Signs that a tool like Puppeteer, Playwright, or Selenium is driving the browser. | Even if you hide the navigator.webdriver flag, anti-bot scripts can plant a trap in console.log that only fires when Chrome's DevTools Runtime is listening. |
| 4. Device hardware | How the device's graphics card (GPU) and installed fonts draw text and images. | Software can mimic a browser, but hardware shows through. Even on a patched Redroid Android container on a server, missing motion sensors and the server's GPU give it away. |
To see how much real hardware varies, we analyzed 13,485 real Windows PCs. While 92% of them shared the exact same network handshake, they produced:
- 522 different graphics renderers (no single GPU appeared on more than 5% of machines).
- 866 different canvas drawings of the exact same text.
- 5,291 unique font sets (1 in 3 machines had a font list no other computer shared).
In 2016, bot detection was a binary wall: you were either blocked or let in. Today, detection is a credit score. Anti-bot systems combine all four layers above with your session history and behavior - how your mouse moves, where you click, and how fast you type.
How platform engineering has adapted
Because there are 10 to 100 times more strategies to choose from - and customers still expect a response in milliseconds - we can no longer try proxies one by one in a slow waterfall. Rebuilding our platform for this reality has meant six changes:
- Intent: Moving from how-based APIs (where a user tells us to send a specific GraphQL payload) to intent-based APIs for both humans and agents (where the user asks to set a zip code and our system chooses whether to use GraphQL, network capture, or a browser).
- Context: Giving the agent the right context across three areas - the user's intent, our internal system state (available resources and connections), and the target website's technology stack.
- Communication: Supporting a team-of-agents topology with agent registration and strict schemas (Protobuf, JSON, or structured Jira payloads) so agents communicate through firm interfaces.
- Tools: Every team at Zyte now runs its own agents, communicating through strict interfaces. Additionally we are building tracing tools directly into Chrome so our agents can reverse-engineer protections and patch browsers automatically.
- Depth: Building a layered cake of strategies that agents can fall back to - from low-cost, low-latency clients all the way down to physical device farms for the hardest websites.
- Control Plane: Building observability, steerability, and security directly into the platform so we can give agents an open canvas to work with safely.
The user is changing: One human decision, N agent requests
For forty years, we built the web for human eyes. That is no longer the majority of traffic. According to Cloudflare Radar data from October 2026, 64.1% of all web page requests are automated.
Until recently, bots did three jobs: training AI models, building search indexes, or feeding real-time APIs. Now there is a fourth job: agents acting on behalf of a person.
I saw the need for this firsthand in Dubai Marina, where my wife and I had to furnish a bare two-bedroom apartment for Airbnb. Using ChatGPT and Gemini to create a design brief was easy; getting it built was exhausting. Here is how doing it the old way compared to what a personal agent can do when it can read the web:
| Step | The old way (human procurement) | The agentic way |
|---|---|---|
| Research | Manually reading dozens of tabs, forums, and vendor reviews over several weeks. | The agent reads 1,184 product pages and checks reviews across 48 suppliers in minutes. |
| Quotes & stock | 8 contractor quotes ranging from AED 37,669 to AED 171,761, with vague delivery dates. | Filters for items in stock (212 of 240) that can be delivered before the deadline. |
| The decision | Severe decision fatigue comparing materials from laminated MDF to Spanish marble. | "Pinterest meets Tinder": The agent presents 4 complete, priced room looks. We swipe on the style we like and buy via Agentic Commerce Protocols (ACP/UCP). |
As AI moves from chatbots (1 web request) to copilots (a few requests) to agents (thousands of requests) and ambient AI working quietly in the background, a new rule applies: the less the human explains, the more the agent reads. One human decision will routinely trigger 10,000 or more agent requests.
Why today's web architecture can’t carry the load
If agent traffic grows by 100 to 1,000 times, and getting each request through anti-bot fences costs 100× more in compute and proxies, we face a 10,000 to 100,000x cost explosion.
Sending full web pages built for human eyes to software agents is massively wasteful.
I tested fetching the exact same product data from a 48-item listing page using three different methods:
| Method | What travels over the wire | Relative cost |
|---|---|---|
| 1. Real browser (Chrome) | Full HTML, CSS, images, and JavaScript rendered for human eyes. | 8.3x |
2. Plain HTTP client (Rust wreq) |
Raw HTML page without running the browser. | 1x (baseline) |
| 3. Agent-ready facts (Cap’n Proto) | Only the product facts, structured and compressed in a binary format for machines. | 0.0065x |
Fetching facts through a full browser is 1,300 times heavier than sending structured data built for an agent. Lightweight browsers like Lightpanda cut browser overhead by 3 to 5x, which helps today, but long term we have to change how websites serve and serialize data.
The challenge is that the web moves at the speed of its defaults. When we scanned the top 100,000 websites Zyte accesses most, 63.3% still did not support HTTP/3 and QUIC - four years after it became a standard. Even among the 36.7% that did, 29.4% came from a single CDN turning it on by default.
The new fences - and keeping the roads open
New rules for agents are being written right now:
- IETF Web Bot Auth: A standard for bots to cryptographically prove their identity.
- Mozilla PACT: A framework for publishers to control which agents can access their data.
- The "Two-Tiered Web": A warning from the OpenID Foundation that the web could split into invited lanes and blocked lanes.
In the 1990s, User-Agent and robots.txt simply asked a bot its name. The new standards ask an agent to prove: "I am acting on behalf of this person, with this authority."
Commercial standards like Model Context Protocol (MCP), WebMCP, and Shopify’s MCP tools will help agents buy products. The danger is who gets to vouch for an agent. If only three or four tech giants are trusted to issue agent credentials, the open web becomes a walled garden. Startups building specialized hardware - like Y Combinator-backed skin-diagnostics company Lumera, or robotics teams in construction, logistics, mining, and energy - need open web data just as much as big chat platforms do. You cannot plan what you cannot see.
What gives me optimism is the strength of our industry.
Alongside established leaders like Zyte, Bright Data, and Oxylabs, dozens of startups backed by Y Combinator, Andreessen Horowitz, and Warburg Pincus - including Exa, Nimble, Tavily, Firecrawl, Byteful, and Massive - are building the next generation of web access tools. Together, we have the engineering experience to take a seat at the table at the IETF and W3C and help write fair rules.
Oliver Strange, who wrote those Sudden cowboy novels on my grandfather’s shelf, was an Englishman who never set foot in America, let alone Texas. He was able to bring the frontier to life for readers around the world because he had open access to information. That access mattered a century ago, and it matters even more for the agents that will act for us tomorrow. Keep building, get involved in the standards, and keep the roads open.
