Zyte Blog
Field notes from the world of data extraction.
Articles, interviews and analysis on how data is gathered, used and fought over — written by the people closest to it.
Browse
Hot topics
Latest
What we've been publishing
Need for speed: How data inertia kills business ambition
Slow web data pipelines can hold back new markets, conversion gains and product launches.
Scrapy vs BeautifulSoup vs Selenium vs Playwright: which should you use?
What each Python scraping tool does, where it fits, and how to choose, with the same scraper written in each and a simple decision guide for beginners.
The web's layered defences - how access barriers combine
Individual barriers may get the headlines, but it's stacked access control mechanisms that pose the biggest block.
How I use Zyte MCP to plan web scrapers
Zyte MCP lets my AI agent investigate a site before I write the scraper, so I can decide where the data should come from, compare an API with the rendered page for missing fields, and end up with a crawl plan I can act on.
Self-healing spiders: letting an agent fix your Scrapy crawl, with a lid on it
Two demos from our Extract Summit Austin workshop: an agent repairs a Scrapy spider after a site redesign, then heals a live crawl mid-run through Spidermon and the Scrapy MCP, with the measured costs and the mistakes that shaped the design.
The anti-ban arms race: Agentic traffic and the fight for the open web
As AI agents transform how we interact with the web, anti-bot systems threaten to fence off the digital open range.
TLS fingerprinting, the handshake that reveals your true intent
TLS fingerprinting fires before you have even asked for a page, and tells you nothing when it does. The least common barrier, and the hardest to diagnose.
How to convert a website to Markdown for AI: 7 methods compared
Seven realistic ways to turn web pages into clean Markdown for AI - what each method covers, where each breaks, and how to choose for training, RAG and agent pipelines.
Train AI on industry-leading Markdown with Zyte’s new extraction mode
Zyte API can now extract a web page’s main content directly as clean, structured Markdown for AI systems, RAG pipelines and developer workflows.
JavaScript renders almost half of sites invisible
JavaScript renders 40.6% of top sites unreadable to a plain HTTP request, not by design but as a side effect of how the modern web is built.
Five takeaways from Extract Summit 2026 Austin
AI is making web scraping faster - but dependable data takes more than working code. Quality, provenance, self-repair, access and responsibility are the new edge.
Rate limiting is a roadblock, not a speed limit
Rate limiting sounds like a speed limit, but 93.7% of rate-limited sites block on the very first request. It is an identity check, not a throttle.
Bluesky
X
Mastodon
LinkedIn
Facebook
dev.to
RSS feedZyte YouTube
Watch & learn
Video · Zyte YouTube
AI generated these Scrapy projects - why I won't ship them
July 7, 2026
Video · Zyte YouTube
Screenshot webpages with this Claude Skill and Zyte API
March 6, 2026
Video · Zyte YouTube
0% Hallucination? RAG + Web Scraping (Step-by-Step)
March 5, 2026
Video · Zyte YouTube
Generate HTML Parsing code the right way with Scrapy & Web Scraping Copilot
February 23, 2026
Video · Zyte YouTube
Zyte API Sessions - flexible cookie management maintaining control
February 6, 2026
Events & community
