Large Language Models (LLMs)

Articles from the Zyte blog about Large Language Models (LLMs).

How do you manage the context window in Claude Code and other coding agents - like a ninja
AI

How do you manage the context window in Claude Code and other coding agents - like a ninja

Claude Code rarely runs out of context now, but you still pay for what you carry. 18 habits checked against the docs: /compact, /clear, caching, MCP.

Ayan Pahwa
A stream of web pages flowing from a globe through a gateway into a vector database
Data gathering for AI

Web Scraping for RAG: Build a Reliable Pipeline with Zyte API, Qdrant and LangChain

Build a RAG pipeline that survives bot blocks and JavaScript pages: fetch LLM-ready Markdown with Zyte API, gate bad pages, and test it before deploying.

Ayan Pahwa
Seven HTML-to-Markdown methods compared
Data gathering for AI

How to convert a website to Markdown for AI: 7 methods compared

Seven realistic ways to turn web pages into clean Markdown for AI - what each method covers, where each breaks, and how to choose for training, RAG and agent pipelines.

Robert Andrews14 min read
Transform HTML pages into Markdown
Data gathering for AI

Train AI on industry-leading Markdown with Zyte’s new extraction mode

Zyte API can now extract a web page’s main content directly as clean, structured Markdown for AI systems, RAG pipelines and developer workflows.

Valter Sciarrillo8 min read
A toy robot walking toward a blocking robots.txt file
Data gathering for AI

Four in 10 sites block AI bots with robots.txt

Nearly four in 10 of the world's top sites are now closed to well-behaved AI crawlers. Here's what operators actually do with robots.txt, and which AI agents they block or welcome.

Robert Andrews
An automatic Claude writer device, writing code.
AI-assisted data extraction

Claude Fable 5 is the new best model for writing scrapers

We ran nine models, including the new GPT-5.6, through our Zyte Scraping Code Benchmark. Claude Fable 5 puts Sol and others in the shade - but it’s pricey, and the best extraction code still depends on the best infrastructure.

Konstantin Lopukhin8 min read
Harness Engineering, part 2: harnessing a data extraction agent
Large Language Models (LLMs)

Harness Engineering, part 2: harnessing a data extraction agent

Point an agent at a website, name the fields, get structured records back. Harness engineering part 2: the harness decisions that make a data extraction agent reliable.

Ayan Pahwa8 min read
Harness Engineering, part 1: what is an agent harness and why it matters
Large Language Models (LLMs)

Harness Engineering, part 1: what is an agent harness and why it matters

Same model, same weights, zero retraining — LangChain changed nothing but the scaffolding around a coding model and jumped it from 30th place to the top five on a benchmark. That scaffolding has a name: the harness. And it's the part you actually control.

Ayan Pahwa8 min read
The harness matters more than the model - Podcast EP07
AI-assisted data extraction

The harness matters more than the model - Podcast EP07

"The model is the engine — but the harness is everything else." In Episode 7, we dig into why the infrastructure layer around your AI model matters more than the model itself, rank the best models available right now, and ask whether the open-weighted revolution is about to make frontier subscriptions obsolete.

John Rooney11 min read
C N T 1216 Main
AI

The best agent skill is the one that says the least

More instruction, worse output. Zyte's head of R&D on why telling your agent exactly what to do can blind it to the obvious answer.

Neha Setia Nagpal14 min read
Why I'm adding GLM-5.2 to my agentic coding arsenal
Large Language Models (LLMs)

Why I'm adding GLM-5.2 to my agentic coding arsenal

Ayan Pahwa ran GLM-5.2 against Claude Sonnet 4.6 and Opus 4.8 on real scraping work: parity on extraction, 8.5x cheaper on planning, slower and wordier.

Ayan Pahwa14 min read
My personal agent setup: the architecture that runs my household and my DevRel work
AI

My personal agent setup: the architecture that runs my household and my DevRel work

"Four people, four diets, two work schedules, and a baby who answers to nobody. That's what finally made me build a personal agent." A walkthrough of the actual architecture I run to hold my household and my DevRel work together — profiles, skills, memory, and the web-data layer that makes it all reach the live web.

Neha Setia Nagpal11 min read

More articles on Large Language Models (LLMs)