Author
Ayan Pahwa
Developer Advocate
Ayan is a developer advocate at Zyte. Ayan writes hands-on, personal-project-driven content about applying AI agents and LLMs to real scraping problems — his "Harness Engineering" series explains what an agent harness is and how to build one for data extraction, and he documents his own agentic coding setup (Claude Code, multi-agent orchestration) and unusual builds like running a scraper on 22 KB of RAM or using an old Raspberry Pi for live trading data. He also writes practical /learn/ guides on data quality tooling (Spidermon), modern HTTP libraries, and production Scrapy/Docker/PostgreSQL setups. Across 25 articles, he blends developer-community coverage (Zyte's first meetup, Scrapy extensions) with practical AI-scraping experimentation.
The page your agent scrapes is now an attack surface. Is it ready for the hostile web?
Real web is becoming hostile for AI Agents. The page your agent scrapes now could be a potential attack surface. Read more and join Zyte's virtual meet-up to see it in action.
Fable 5.1 shipped, and GLM-5.3-Flash turned out to be someone I already met
A personal take on Claude Fable 5.1 and GLM-5.3-Flash: real benchmarks, a live extraction test, and the model I guessed before Zhipu confirmed it.
Web data in a reactive notebook: an introduction to marimo
marimo is a reactive Python notebook that reruns only affected cells. Learn to scrape web data with Zyte API, chart prices, and cache costly API calls.
WebFetch is ‘lossy’ by design: give your coding agent a better fetch in one command
Your coding agent built-in webfetch tool is not the best and it's hampering your research and coding workflows, fix it with one CLI tool and never face blocks again.
Harness Engineering, part 4: giving your agent a custom fetch tool that survives the real web
Your AI agent is as powerful as the tools it has access to. Here's a tutorial on how you can create your own custom agent tool using Claude Agent SDK for zyte which makes getting structured data from web a breeze.
I joined the Real Python podcast to talk harnesses and Scrapy
I was recently a guest on the Real Python podcast where host Christopher Bailey opened with the question that has been following me around all year: which matters more, the model or the harness around it? Let's look into it.
A guide to Scrapy item types
Scrapy supports multiple item types, but which should you use, and why.
Harness Engineering #3- Headless mode: the minimal agent harness
What's the smallest harness that still works? Turns out it's already sitting inside almost every coding agent you have installed — headless mode: same loop, tools, and reasoning as the interactive agent, minus the human in the chair. We point it at a web page and pull clean, structured data out the other end in about ten lines.
GPT-5.6, Fable 5, and GLM-5.2 entered a bar “crawl” and got hit by The Rate Limit
When OpenAI shipped GPT-5.6 and Anthropic's Fable 5 was sitting at the top of the price list, my question was not "which one wins a leaderboard." It was the one I actually pay for: for the scraping I do, how much model do I need to buy?
Harness Engineering, part 2: harnessing a data extraction agent
Point it at a website, tell it which fields you want, get back clean structured records. That's the agent we're designing in this post — and the interesting part isn't the model, it's the harness decisions that make it actually reliable at scale.
Harness Engineering, part 1: what is an agent harness and why it matters
Same model, same weights, zero retraining — LangChain changed nothing but the scaffolding around a coding model and jumped it from 30th place to the top five on a benchmark. That scaffolding has a name: the harness. And it's the part you actually control.
Zyte's first Developer Community Meetup: the recap, slides, and recording
AI agents can now write, run, and self-heal your web scrapers, and in Zyte's first-ever Web Scraping Community Meetup we show you exactly how. Live demos, a Claude Code plugin that turns a prompt into production-ready data, and a fireside chat on where AI is really heading.
How to run any model inside Claude Code
I run GLM 5.2 inside Claude Code with the same tools, the same skills, and the same agent loop. Three environment variables is all it takes.
Why I'm adding GLM-5.2 to my agentic coding arsenal
Is GLM-5.2 really closing the gap to Anthropic - and at just a fraction of the cost - or is it just more AI hype? I think so, and let me show you why.
How to build your first Scrapy extension
My Scrapy project now plays a triumphant fanfare when a crawl finishes clean and a sad trombone when it doesn't, which is also how I finally learned what Scrapy's extension points are actually for.
Now what exactly is loop engineering? And where do Anthropic's Fable 5 model and web scraping fit in?
A viral clip from Claude Code's creator put a name to something a lot of us have been circling: loop engineering. Here's why web scraping may be its best-fit domain — and what that means in practice.
Web scraping on an iPhone? Yes, really!
When you can scrape the web by API, a world of possibility opens up. Yes, you can extract live web data using iOS Shortcuts.
Automate deployment of your web scraper on VPS with Ubuntu 24.04 cloud-init
Your VPS is ready, but now you need to work through the same sequence you have run a dozen times before: apt update, apt install python3-pip, pip install scrapy, playwright install chromium, the Chromium dependency list that never installs cleanly on the first try, Redis, possibly Postgres, whatever else this particular project needs.
Web scraping on 22 KB of RAM: Fitting the world on an ESP8266 microcontroller
Data-gathering doesn’t have to be memory-intensive. You can fit the world’s weather on a 9cm-square board, when you move the work to a web scraping API.
Flatcar Linux for web scrapers: deploy immutable containers with just one config file
The next time you spin up a VPS to give it a persistent home, you spend the better part of an afternoon rebuilding from memory. Here's a tool to help using Flatcar Linux
My agentic coding setup: Claude Code, multi-agent orchestration, and how I actually work
Ayan's 4 agent team, using Claude's /goal, and the models and coding agents he uses to code effectively.
OpenClaw and Claude helped me buy the perfect sneakers using Zyte API
Quickly compare e-commerce products across any site with an agent, a skill and an AI-powered web scraping API.
Giving spidey-senses to your web scraping spiders using Spidermon
Learn how Spidermon helps you monitor web scraping data quality in real time. Validate items, track field coverage, and get alerts before bad data impacts your pipeline.
How to ensure data quality in your Scrapy web scraping projects using Spidermon and Claude Code
Spidermon is an open-source monitoring framework for Scrapy. You attach it to your spider, define what "success" looks like, and it automatically checks your crawl results after the spider closes, flagging anything that doesn't meet your standards.
Why your API responses look like gibberish: the gzip decompression trap
The script was working. Requests were going out, responses were coming back with HTTP 200. But the response body was unreadable noise, a wall of binary characters that crashed the JSON parser and reported "no data found". No error code, no timeout, no network failure; just garbage where structured data should be.
Code is cheap, show me the talk: How copilots are re-engineering developers
Discover how AI copilots like Zyte’s Web Scraping Copilot are transforming developer workflows—making code a commodity and shifting value to problem-solving and prompting skills.
Build your own MCP server: LLMs meets web data with Zyte API
Learn how to build your own Model Context Protocol (MCP) server to connect LLMs with real-time web data using Zyte API, FastMCP, and the Docker MCP toolkit.
Stop using Python requests for web scraping: Use these modern modules instead
While the 'Requests' library remains the default choice for many Python developers due to its reliability and extensive documentation, the Python HTTP landscape has evolved considerably. Modern alternatives now offer significant advantages, including built-in asynchronous support, HTTP/2 compatibility, enhanced performance, and up-to-date TLS handling.
Brewing a bot: RAG and web data fuel the perfect coffee recommendation
Learn how to build a real-time AI chatbot using RAG, web scraping, Zyte API, LangChain, and OpenAI. Scrape JavaScript-heavy websites, store data in a vector database, and generate accurate answers from fresh web data.
Building a production-style web scraper with Scrapy, Docker, and PostgreSQL
Demo project scrape2postgresql shows how to scrape structured data with Scrapy, store it in PostgreSQL, and run both the spider and database in separate containers using Docker Compose.
Scraping Swiss Army Knife: My personal fix for web setup fatigue using Docker, Scrapy and Zyte
Tired of repeating web scraping setup? Learn how a multi-arch Docker container with Scrapy, Zyte, Requests, and Pandas speeds up exploration and debugging.
How I trade gold using e-ink, live data and an old Raspberry Pi
Track real-world gold and silver retail prices automatically using Zyte API, Python, and a Raspberry Pi with an e-ink display. Learn how to scrape rendered HTML, parse prices, and build an always-on trading dashboard.