Neha Setia Nagpal

Treat your AI skills like software, starting with evals

Most AI skills are never tested — and it shows. Here's how Zyte evaluates scraping skills like real software, catching failures demos miss.

The best agent skill is the one that says the least

More instruction, worse output. Zyte's head of R&D on why telling your agent exactly what to do can blind it to the obvious answer.

My personal agent setup: the architecture that runs my household and my DevRel work

"Four people, four diets, two work schedules, and a baby who answers to nobody. That's what finally made me build a personal agent." A walkthrough of the actual architecture I run to hold my household and my DevRel work together — profiles, skills, memory, and the web-data layer that makes it all reach the live web.

What's becoming of web scraping developers in the age of AI agents?

AI agents can generate code, suggest selectors, and draft crawl logic. What they can't do is design the system that decides when to stop, what to trust, and how to recover when the web pushes back. That job still belongs to a human.

What multi-agent orchestration looks like in a large-scale web scraping project

Multi-agent orchestration is having its moment. The diagrams are everywhere now. Boxes for planners, boxes for hands, boxes for daemons, arrows to a shared brain, a human floating at the top. They keep getting prettier. The part where the web pushes back is still the part nobody draws.

NotAnInterview: “I Have Superpowers Now"

The problem was a project with 12,000 websites to crawl, and there’s no world where you write custom spiders for 12,000 websites, not with a human team and certainly not sustainably. So Javier built a workflow: a set of AI prompts that could analyze a website, figure out its structure, and generate a crawl configuration that a generic spider could then use.

I'm not the same developer I was before LLMs

I've been running a series of conversations with developers at Zyte to understand what's actually changed in the way they work since LLMs showed up. Not the headlines. The day-to-day. What they delegate, what they don't, what they notice, what surprises them. This one was different on two counts.

AI won’t fix your data quality (until you answer these three questions)

In our interview, a QA expert warns - before you delegate web scraping quality assurance to AI, make sure you can describe what ‘good’ looks like for yourself.

From screenshot to shopping list in 90 seconds

I built a mood board pipeline that starts with a screenshot. Claude Skills and Zyte API for search, product extraction, and image embedding at any scale.

The Scrapy whisperer: Adrian Chaves on Web Scraping Copilot

An interview with Scrapy maintainer Adrian Chaves on Zyte’s Web Scraping Copilot, AI-generated parsing code, and building reliable scraping workflows.

More data, more trouble: How a perfect corpus corrupted my AI dream

A failed AI experiment reveals why adding more data doesn’t always improve LLM outputs. Learn when web scraping, RAG, and curated datasets actually make AI better.

Claude skills, MCP or Web Scraping Copilot: Which should you choose?

Compare Claude skills, MCP servers, and Web Scraping Copilot to understand when to use each for AI-powered web scraping, data extraction, and production pipelines with Zyte API.

A Deep Dive into Zyte's Open-Source Libraries

Discover how Zyte’s open-source libraries like ClearHTML, Extruct, Chomp.js, and more simplify web data extraction and processing.

Analyze web data quickly with Jupyter Notebooks and Zyte API

With AI Scraping in Zyte API, you can pull data from any e-commerce website straight into your Jupyter notebooks.

Overcoming web scraping challenges of Puppeteer and Playwright

Explore issues like browser farm management, IP rotation, and anti-scraping measures that can complicate large-scale operations.

How Session Management Minimizes Bans and Enhances Data Quality in Web Scraping

Learn how managing user sessions in web scraping can help overcome website bans, handle IP rate limits, streamline cookie management, and avoid detection.

Best web scraping methods for JavaScript-heavy websites

Explore different web scraping methods, from replicating JavaScript requests to using browser automation tools like Playwright, Puppeteer, and Selenium. Learn the pros and cons of each approach and how to scale your web scraping projects efficiently.

Selenium, Puppeteer, Playwright: Which tool is right for web scraping at scale?

Discover the strengths and limitations of Selenium, Puppeteer, and Playwright for web scraping. Learn about their scalability challenges and what to consider when choosing the right tool for your scraping needs.

Why are Sessions Crucial in Web Data Extraction?

Sessions help maintain state, handle cookies, boost efficiency, and bypass anti-scraping measures, making your web scraping projects more robust and effective.

4 essential Scrapy plugins for building efficient and effective spiders

Here are four essential Scrapy plugins we use to build efficient web crawlers for our customers.

The Scraper’s System Part 2: Explorer’s Compass to analyze websites

Learn about the scrapers system: Explorer’s Compass to analyze websites.

How Zyte API takes care of the fundamental needs of your web scraping project!

Zyte API is a fundamental for web scraping, it eliminates the most time-consuming and difficult challenges of scraping.

How Web Scraping and Graph Databases Can Power Recommendation Engines

I recently had the pleasure of participating in the third episode of Graphversation, a monthly live stream series that brings together graph experts and Neo4j enthusiasts for engaging and enlightening discussions about the captivating world of graphs.

Superpower your Smart Proxy Manager with new headless browser libraries

Scraper's System: Secrets to architect scalable web scraping

A scalable web scraping project is a complex system with dynamic situations that consist of changing problems that interact with each other.

How To Use Selenium With Zyte Smart Proxy Manager

Learn how to use Selenium With Zyte Smart Proxy Manager. Our library provides easy integration and management of headless capabilities.

How to use Playwright with Smart Proxy Manager

Integrate Playwright with Zyte Smart Proxy Manager for efficient web automation.

The best way to Use Puppeteer With Zyte Smart Proxy Manager

Good news for all the Puppeteer users who are looking for an easy-to-integrate anti-ban solution for extracting data from Javascript-heavy websites.

Developer’s guide to rotating proxies in Python

Correct use of rotating proxies is a key ingredient of web data extraction. In this guide, you will learn how to set up IP rotation for scraping in Python.

What is a rotating proxy? Why is IP rotation important for scraping?

In this article, we help you understand what is a rotating proxy and why you need rotating proxies for web scraping.

Find the best pint using web scraping on St. Patrick’s day

“Who serves the best pint of Guinness?” Read our St Patrick's Day Special: Finding the best pint of Guinness in Dublin using web scraping and data science.

Why you need the best proxies for web scraping

Proxies for web scraping are very important. Depending on the website you are scraping, select between data center proxies, residential proxies, and more.