PINGDOM_CHECK
Senior editor

Robert Andrews

Robert is a journalist and editor turned content strategist who eats and sleeps the web. Previously senior editor at Google, News Corp, ContentNext and others.

The latest from Robert Andrews

Blog post thumbnail
Leadership

Death of the Proxy? There’s An API for That

The proxy era is ending as web scraping shifts from managing IP pools to smarter, API-driven solutions.
Robert Andrews
August 26, 2025
Blog post thumbnail
Leadership

Why the Best Engineers Are Actually Lazy

There’s a popular archetype in web scraping circles: the heroic engineer who fights CAPTCHAs at 3 a.m., hand-tunes a proxy farm before breakfast, then rewrites four spiders after lunch because the target sites pushed new JavaScript.
Robert Andrews
August 18, 2025
Blog post thumbnail
Open Source

The future of Scrapy: Smarter, faster and ready for AI-powered scraping

What does the future hold for the tool some describe as “the gift that revolutionised web scraping”?
Robert Andrews
June 23, 2025
Blog post thumbnail
Open Source

Rise of the Data Vendor: How Outsourcing is Transforming Supply and Fuelling Businesses

With the emergence of managed data extraction vendors, businesses no longer need to gather web data themselves.
Robert Andrews
June 20, 2025