What happens to web data when a machine can write the scraper, investigate the failure and help fix it?
At Extract Summit Dublin, that question ran from hands-on workshops to the main stage. The answers from experts who spoke were practical: look closely at the page, measure what you are paying for and give automated systems a way to check their work.
As Suzanne Hassett, chief operating officer at Zyte, put it:
“The moat has moved. When a machine can read almost anything, what’s still hard? Where does the advantage sit now?”
Here are five answers speakers provided over two days of web data engineering talk.
1. Cheaper code doesn’t make the web free
Writing a spider is only part of the job. Teams still pay to fetch pages, manage infrastructure, respond to changes and deliver data that can be trusted.
Zyte CEO Shane Evans framed the commercial question:
“If you run a scraping team and you want to cut the bill next year, where would you put your effort?”
The answer may not be another improvement to code generation. It may be the less glamorous work of understanding where the operating costs actually sit.
Robert Vähhi, co-founder and chief technology and product officer of TrustSig, offered one reason to examine suppliers closely. In a study of proxy products, his team observed substantial overlap in exit IP addresses across providers. Some highly overlapping pools had list prices up to 10 times apart. As he put it: “One brand lists the same addresses at 10 times another brand’s price.”
Overlap alone doesn’t mean the products are identical: performance, reliability and sourcing still matter. But it does make price, and what sits behind it, worth investigating.
The first day’s workshops offered a complementary lesson from Fernando Tadao Ito, team lead at Zyte:
“Use the right tool for the job - in the code, whether you write it or an LLM does.”
Faster code is useful; choosing the right approach still matters.
2. Browsers don’t all need to do the same job
A conventional browser is built to display a page to a person. For automated data collection, drawing every pixel may be unnecessary work.
Pierre Tachoire, co-founder of Lightpanda, presented the headless browser built for automation without a graphical interface. “Traditional browsers weren’t built for automation. Lightpanda was,” he said. In a benchmark shared in his presentation, it used considerably less memory and completed the test faster than Chrome. The result is specific to that test, not a guarantee for every site.
Amit Parpara, engineering team lead at Tavily by Nebius, explored a related idea: browserless scraping that executes JavaScript without the full rendering pipeline. The trade-off is important. Content dependent on canvas or WebGL, sustained interaction and interactive challenges may still need a full browser.
“Understanding the boundaries of browserless web scraping is crucial,” Parpara said. The question is not whether to abandon browsers, but when a lighter tool is enough.
3. Access is about sessions, signals and identity
Ionut Ciubotariu, an antiban engineer at Zyte, drew this lesson from a year working on web access: sites don’t judge requests in isolation. They can evaluate a session’s history and whether its network, browser and behavioral signals make sense together.
His advice extended to understanding the page itself:
“Read what the page does first. Sometimes it ships its own answer.”
In other words, before adding complexity, investigate what the site is actually returning and why.
Pierluigi Vinciguerra, co-founder and CTO of Databoutique and owner of The Web Scraping Club, examined a related tension for publishers.
He said AI makes it cheaper to analyze anti-bot code, but it also brings a new kind of visitor: an agent acting for a customer:
“If agents become real revenue, the job is a balance: open to agents, closed to bots. But to let agents in and keep bots out, an antibot first has to know who is in front of it.”
That is a harder distinction than human versus machine. A useful agent and a fraudulent bot may both be automated; intent, identity and context matter. The question of agent identity also came up at Extract Summit 2025.
4. A self-maintaining spider needs memory and checks
“Building a spider is one pass. Keeping it working is a loop,” said Iván Sánchez, senior data scientist at Zyte.
His presentation explored giving coding agents a website knowledge base: a record of pages, working spiders, site-specific quirks and previous runs. Monitoring can detect changes, then propose and test repairs, with a person approving them or an automated path applying them.
In an early test across four e-commerce sites, the approach reduced time and cost for one coding-agent setup while improving its measured quality. Results varied by setup, and the sample was small. Still, the direction is compelling: the agent needs more than instructions to write code. It needs evidence about the site and a way to decide whether a proposed fix works.
That makes maintenance a system-design problem, not simply a prompting problem.
5. Public data becomes more revealing when connected
Domagoj Marić, developer advocate at Zyte, closed a different kind of loop: the one between scattered public information and personal risk.
He warned:
“Everything an attacker needs, is already published, mostly by the target.”
Usernames, photos, job details and other fragments can be connected into a profile convincing enough to support a phishing pretext or impersonation.
Each piece may look harmless alone; the combination is what changes the picture.
That is a useful reminder for an industry built around turning web fragments into structured information. The ability to collect and connect data is powerful. How that capability is used, and the boundaries around it, matter just as much.
Parting thought
The moat has moved. After two days in Dublin, the advantage looked less like the ability to write a scraper and more like knowing how to make the whole operation work.
So, the theme was not so much that AI has solved web scraping - if only.
Rather, it has changed where people need to apply their judgment: to the economics, the system around the code and the consequences of the data it produces.
