AI

Articles from the Zyte blog about AI.

Group on stage at Extract Summit Dublin 2026
Web data collection

Five takeaways from Extract Summit Dublin

AI is making scraping code cheaper to write. Extract Summit Dublin explored the harder work that remains: access, infrastructure, maintenance and trust.

Robert Andrews
How do you manage the context window in Claude Code and other coding agents - like a ninja
AI

How do you manage the context window in Claude Code and other coding agents - like a ninja

Claude Code rarely runs out of context now, but you still pay for what you carry. 18 habits checked against the docs: /compact, /clear, caching, MCP.

Ayan Pahwa
What is a self-healing web scraper?
AI-assisted data extraction

What is a self-healing web scraper?

A self-healing web scraper detects when a site change breaks extraction, repairs the code within limits you set, proves the fix with tests, and escalates when it can't. What it fixes, what it doesn't, how the loop works and what a repair costs.

John Rooney
self-healing-spider-agent
AI-assisted data extraction

Self-healing scrapers: four things to put in place before an AI agent fixes your spiders

What decides whether a self-healing scraper's repair is any good: detection during the crawl, a specific failure report, extraction code the agent can target, and tests that can fail.

John Rooney
A stream of web pages flowing from a globe through a gateway into a vector database
Data gathering for AI

Web Scraping for RAG: Build a Reliable Pipeline with Zyte API, Qdrant and LangChain

Build a RAG pipeline that survives bot blocks and JavaScript pages: fetch LLM-ready Markdown with Zyte API, gate bad pages, and test it before deploying.

Ayan Pahwa
12 things I use Zyte MCP for, ranked, with prompts and rules you can copy
AI

12 things I use Zyte MCP for, ranked, with prompts and rules you can copy

What can you use Zyte MCP for? 12 uses, 11 tested, ranked, each with a copy-paste prompt, a CLAUDE.md rule, and a skill idea.

Ayan Pahwa
How I use Zyte MCP to plan web scrapers
AI

How I use Zyte MCP to plan web scrapers

Zyte MCP lets my AI agent investigate a site before I write the scraper, so I can decide where the data should come from, compare an API with the rendered page for missing fields, and end up with a crawl plan I can act on.

John Rooney
A spider web inside a dashed loop, with one broken strand being mended by a glowing magenta thread
Tool-assisted coding

Self-healing spiders: letting an agent fix your Scrapy crawl, with a lid on it

Two demos from our Extract Summit Austin workshop: an agent repairs a Scrapy spider after a site redesign, then heals a live crawl mid-run through Spidermon and the Scrapy MCP, with the measured costs and the mistakes that shaped the design.

Ayan Pahwa
Akshay Philar speaking at Extract Summit about the anti-ban arms race and the open web.
Anti-ban

The anti-ban arms race: Agentic traffic and the fight for the open web

As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.

Akshay Philar9 min read
Seven HTML-to-Markdown methods compared
Data gathering for AI

How to convert a website to Markdown for AI: 7 methods compared

Seven realistic ways to turn web pages into clean Markdown for AI - what each method covers, where each breaks, and how to choose for training, RAG and agent pipelines.

Robert Andrews14 min read
Transform HTML pages into Markdown
Data gathering for AI

Train AI on industry-leading Markdown with Zyte’s new extraction mode

Zyte API can now extract a web page’s main content directly as clean, structured Markdown for AI systems, RAG pipelines and developer workflows.

Valter Sciarrillo8 min read
Zyte MCP chat box connected to fetch, search and extract tools.
Data gathering for AI

Introducing Zyte MCP: Powerful web data gathering for your agent

Zyte MCP brings live web access, structured extraction and Scrapy Cloud operations into coding agents such as Claude Code, Cursor and Codex.

Valter Sciarrillo7 min read

More articles on AI