I joined the Real Python podcast to talk harnesses and Scrapy

I was recently a guest on the Real Python podcast, where host Christopher Bailey and I recorded episode 304, published on Friday, July 24, 2026. Bailey opened with the question that has been following me around all year: which matters more, the model or the harness around it? We got there, but we started somewhere much smaller, with a 75-line Scrapy extension that plays a fanfare when a crawl finishes clean.

It started with a 75-line Scrapy extension

I came to Scrapy late compared with most of my colleagues at Zyte, and my way into anything technical has always been to build something badly before reading the documentation properly. That approach costs me days sometimes, and I keep doing it anyway. Scrapy is a big, opinionated, batteries-included framework with a lot of moving parts, and reading about how those parts hand work to each other was never going to stick for me. So I wrote the smallest thing I could that was forced to touch the internals: an extension that hooks Scrapy's signals, watches the crawl, and plays a triumphant fanfare when the spider finishes with the item count I expected. When it does not, it plays a sad trombone.

That is partly a joke, but when you are scraping hundreds or thousands of test items you should not be sitting there reading the terminal, and a sound from a background process tells you how the run ended without you looking. Along the way I got the thing I actually wanted, which was a map of where the signals fire and what the framework does between one scraped item and the next. Scrapy is to scraping roughly what Django is to web development: you write a spider, tell it where to start and which links to follow, and the framework handles the scheduling, the downloading, the pagination, and where your items end up. The extension was never meant to be published, and it only exists because my team ran out of content ideas one week, which is how it reached the PyCoder's Weekly feed where Bailey found it.

Where Scrapy stops and Zyte API starts

Scraping a page is one part of the job. Doing it across many sites, at volume, on a schedule, is where you meet rate limits, blocking, and more. Scrapy is deliberately well behaved about that, and new projects are generated with ROBOTSTXT_OBEY set to True, but the framework is not in the business of getting you through anywhere with ease. That is where Zyte API comes in, and the part I wanted to land on the show is that it works either way, plugged into a Scrapy project or on its own as a single call:

1curl -s https://api.zyte.com/v1/extract \
2  -u "$ZYTE_API_KEY:" \
3  -H 'Content-Type: application/json' \
4  -d '{"url": "https://books.toscrape.com/", "browserHtml": true}'

The other half of that conversation was selectors. Writing CSS or XPath expressions is the first thing you learn about scraping and the first thing that breaks, because the page you parse today gets redesigned next month and your spider quietly starts returning nothing. Automatic extraction is the way around that for common page types: tell Zyte API the target is a product page or an article, and you get structured fields back with no selector to maintain.

An agent is a model and a harness

Everyone agrees roughly what an AI agent is, which is AI that does things rather than suggesting what you should go and do. The framing I find more useful is that an agent is nothing more than a model plus a harness. The model is the part everyone argues about, Opus or Sonnet or Fable if you are in the Anthropic ecosystem. The harness is wherever you invoke that model from. When you chat on claude.ai, that chat interface is your harness. When Claude Code runs in your terminal, reading files and writing code and checking pull requests, Claude Code is the harness. Claude for Slack is the same model with Slack as the harness.

That gives you a test you can apply for. Swap the model you are using for a different one, and if the task breaks, the thing that needs fixing is your harness. Everything the model can see, the scaffolding around it is the harness: the bash access, the MCP servers, the Python scripts, the tool that fetches a page for it.

The harness job that people notice least is context management. Every API call to a model is stateless, so the model has no memory of what you said a minute ago, and the harness sends the whole conversation on every single call along with information of available tools, this is called Agent Loop. Type "hi" after 10,000 messages and all 10,000 go with it. My longer argument for why this layer is where the durable work accumulates while the model slides toward being a commodity is in Harness Engineering, part 1.

Flirt with all the models, but marry one harness

I said that line off the cuff in a Zyte video once, and it has stuck harder than anything else I have said this year: "flirt with all the models, but marry one harness." My own setup follows it fairly literally. OpenRouter gives me one API key that reaches every model, OpenCode is the harness I reach for first with Claude Code second, and I swap models by task, so Opus plans and Sonnet and Haiku handle execution and file IO. Anything on the web that comes back blocked falls through to Zyte API.

What Bailey found more surprising was everything that is not in there. I do not use memory, I do not define agent personas, and there is no second brain or vault of instructions for my agents to consult. That is a decision rather than a backlog item, because memory cuts both ways and I would rather hand the same instruction to ten different models myself than maintain a store that all ten have to interpret correctly. If I want the current session to plan, I tell it that it is planning now.

What I do keep is a loop. Once I approve a plan, the plan and a checklist get written to disk, and subagents execute against that checklist while their work gets matched back against it and against the tests. That instruction lives in my CLAUDE.md, and it is the piece I would defend hardest, because it survives me changing my mind about every other component. The rest is written up in my agentic coding setup.

Cheap per token is not the same as a cheaper bill

Bailey asked how I keep token use under control, which produced the most concrete example in the episode. GLM-5.2 from Z.ai lists at roughly a tenth of what the flagship models charge per token, so on paper you cut your bill by 90%. In practice the model is extremely thorough and it likes to talk, and it will answer a single message with a couple of thousand words. Multiply that across a working day and the cheaper per-token price can still hand you a bigger invoice, which is why I wrote about adding GLM-5.2 to my arsenal rather than replacing everything with it.

There is a small genre of agent skills that exist mainly to make models stop talking, and caveman and ponytail are the two Bailey and I compared notes on. Ponytail is the one I kept, on the strength of a README about the quiet engineer with the big glasses who says very little and writes the most portable function in the codebase. My advice with both is to take the two or three lines that match how you already work and drop the rest, because the version of this fix that lasts is tuning your own CLAUDE.md or AGENTS.md to the amount of prose you actually want back. Every lab is subsidizing token use right now, and token-maxing gets expensive the day that stops.

Your agent gets blocked too

Every harness ships some kind of built-in web fetch tool, and all of them run from infrastructure that a great many other people are hitting at the same moment, so agents get blocked constantly. They collect 403s, they land on browser checks, and they load pages that need a rendering before there is anything on them to read.

The failure mode is what makes this worth caring about, because it is quiet. Your agent does not stop and report that the research failed. It comes back with a confident, well-organized answer assembled from the pages it could reach, and the page that held the real answer is missing from it, with nothing to tell you what happened. What my team did about it was add a fallback rule to CLAUDE.md:

1When a web fetch is blocked (403 or 429, a block, a challenge page, or an empty JavaScript shell), do not give up and do not ask me. Retry the request through Zyte API and carry on.

That one rule noticeably improved the quality of the research I get back, and it is worth saying that Zyte has since shipped the idea properly, so the hand-rolled version I described on the show is no longer the only option. The Zyte add-ons for Claude Code and other agentic tools give your agent unblocking without you writing the glue.

Also in the episode

Bailey asks his guests about more than their day job, so we ranged past scraping. We got into small local models, including the two-billion-parameter variant of Gemma 4 and why selective weight activation is the development I am watching if you came up through embedded work like I did. My own projects took a back seat, but they are in there: the self-hosted setup I rebuilt from scratch this year on Proxmox and OPNsense, and noalgotube, which exists because YouTube's recommendations started burying the 12 channels I actually follow. On the Python side I am watching async maturity, since so much of scraping is waiting, and Python on WebAssembly. Bailey also got a coffee-roasting confession out of me.

Listen to the episode

Episode 304 of the Real Python podcast runs 58 minutes, and Christopher Bailey put together a thorough set of show notes with links to everything either of us mentioned. If the harness thread is the part you want more of, Harness Engineering, part 1 is where I set the argument out properly, and part 3 gets into running an agent in headless mode, which is the leanest harness you can build.

And if you would rather your own agents stopped handing you half-researched answers, Zyte API has a free trial, and the unblocking runs in the background while you get on with the work.