A conversation with a principal engineer who’s been scraping the web for 15 years.

Before we begin, a disclaimer.

This is not an interview.

Javier said so himself, about four minutes into our call. “I know you said this is not an interview,” he told me, and then he proceeded to give me forty-five minutes of the most honest, unguarded thinking I’ve heard from a developer in a long time.

That’s the format now.

I’ve been having conversations with people at Zyte, not for a product launch but because I’m genuinely trying to understand what happened to developers when LLMs arrived. Not the announcement version. The real version. What changed in the actual work, what got easier, what got harder, and what, if anything, stayed the same.

Javier is a principal engineer who’s been in the web scraping industry for fifteen years. He was at Scrapinghub when there were thirty people and everyone wore four hats, and he’s cycled through delivery, management, architecture, and back again. He currently works on internal AI tooling and leads proof-of-concepts for how the company can use AI to solve problems it’s been staring at for years.

He doesn’t use social media. He reviews code and designs systems. He said yes to this conversation on the condition that it wasn’t an interview.

So. This is not an interview.

First, I asked him to explain his job to an alien.

He laughed, then actually answered.

“I’m focusing on tooling. Creating tools for internal use. And now, working on AI: how to implement it, how to use it inside the company.”

He told me about a context database he’s currently designing. The problem: information spreads across too many sources, and nobody can find what they need when they need it. The solution: an AI layer that can analyze all of it and surface what’s relevant.

That’s the job. Which means his relationship to AI isn’t “I use it to write code faster.” It’s “I’m building the infrastructure other people will use to work with AI.” He lives a level above the tool.

Two years ago, before agents were a product, he built one.

He called it SAM. Spider AI Maker.

The problem was a legacy project with 12,000 websites to crawl, and there’s no world where you write custom spiders for 12,000 websites, not with a human team and certainly not sustainably.

So Javier built a workflow: a set of AI prompts that could analyze a website, figure out its structure, and generate a crawl configuration that a generic spider could then use.

“At that time, I put a workflow with some steps using AI to analyze the website, try to figure out how to crawl and extract information. With that, I created a crawl config that could be used by a generic spider.”

He built this two years ago, before the current wave of agent frameworks and before everyone had an opinion about agentic workflows. He built it because he had a real problem and he solved it the way engineers solve things: whatever works.

I found that detail important. The developers who seem most comfortable with AI right now aren’t necessarily the ones who read every framework release. They’re the ones who’ve been solving real problems long enough to know what a solution should look like before they start building it.

Then he told me about spec-driven development, and I stopped taking notes and just listened.

This is his workflow for building anything with AI now.

You start with a raw idea, but you don’t throw the whole thing at AI and ask it to build. You start a conversation: what do I want to build, how do I want to do it, what are the different approaches? You iterate, disagree, validate.

Then, critically, you document everything: every conclusion, every open question, every decision and why. The AI writes it up, clean and structured and complete.

“Anytime I want to restore that context, I can start a new session from scratch and have the full picture. Instead of having different AI chats that each have a piece of the context.”

He’s solving the memory problem manually, not waiting for a tool to do it for him. He maintains a repository of documents that is essentially a living spec for everything he’s building. New session, new model, doesn’t matter. The context travels with him.

I’ve seen variations of this approach from other developers, but hearing Javier describe it I realized what it actually is: a way of keeping the thinking in your hands. The documents aren’t just notes. They’re proof that you were the one doing the reasoning. AI wrote the summary. You did the thinking.

Halfway through the conversation, he said it.

“I have superpowers now.”

He said it the way someone says something they’ve been sitting with for a while: not a hot take, not a caption, just a genuine observation about his own experience. There was something childlike about it, something lit up and certain, and I wanted to understand exactly what he meant.

So I asked him to name the superpowers.

He paused. And then:

“All the knowledge you have, all the sense of how things are good or bad, what is a good approach or a bad approach. It’s like I know how to do things. And I know there are a lot of parts that are tedious or time-consuming or low-value. Now those can be accelerated.”

These superpowers aren’t new skills. They’re everything he already knew, now moving faster. Fifteen years of judgment. Of seeing bad architecture fail. Of knowing what production-ready actually means. That’s the input. AI is the multiplier.

“AI multiplies what you give it as input.”

That’s his actual thesis, and it has a hard implication: if you bring nothing, AI multiplies nothing. The speed you unlock is proportional to the clarity you bring, and clarity comes from experience.

He also said something that reframed the whole thing for me. He doesn’t think of AI as a tool; he thinks of it as a thinking partner. A buddy you can discuss with. Someone to validate ideas against, catch your blind spots, help you explore before you commit.

“It’s like having a buddy where you can discuss and iterate ideas. Yeah, that’s mostly the main use I’m getting out of it now.”

That distinction matters. A tool does what you tell it, while a thinking partner pushes back. The relationship changes, and so does the output.

I asked him what stays foundational. What’s still true if you take LLMs out of the picture.

I’ve been asking this in every conversation I have with developers. It started as genuine curiosity: what is so intrinsic to how you think and work that no model can touch it?

Javier went quiet for a moment.

“Critical thinking. Validating ideas. Thinking about different options for things. This is how my mind works.”

He mentioned that at the beginning he didn’t trust AI. He reviewed every line of generated code and didn’t delegate anything he couldn’t evaluate. That skepticism wasn’t a failure to adapt. It was taste doing its job.

“I have trust issues with AI that tends to agree with everything. You’re always the smartest, you have the best ideas. I need something to push back. Think in a different way.”

He’s not wrong. The model that always agrees is the most dangerous tool in the room. Javier knows this because he’s spent fifteen years in environments where the wrong architecture costs real time and real money, and you develop a kind of immune system for confident wrongness.

The foundational thing isn’t a skill. It’s a relationship with uncertainty: knowing what you know, knowing what you don’t, and not pretending the difference doesn’t exist.

That’s the thing a junior developer can’t shortcut. You can get the output without the judgment that lets you evaluate it, and for a while you might not know the difference. That might be the real risk here: not that AI replaces developers, but that it makes it easier to skip building the part of you that knows when something is good.

At the end, he got speculative.

He’d been thinking about a standard he’d seen emerging: a way for AI systems to access website content directly, a protocol for structured data access at the web layer. If it became universal, it wouldn’t just change how web scraping works. It could change what web scraping even means.

“If this becomes the standard, you basically have an API for data for any website on the web. That would eliminate the need to do scraping anymore.”

He said it as a genuine question, not a prediction. He doesn’t know. Nobody does. But the fact that a principal engineer with fifteen years in the industry is seriously sitting with this question says something about how fast the ground is shifting.

Here’s what I’m building with these conversations.

I keep asking developers what changed, what didn’t, and what they’re genuinely uncertain about. And I keep finding the same shape in the answers. The ones who seem most at ease aren’t the ones with the newest tools. They’re the ones who know what they want to build before they open the chat window.

That clarity, that trust in your own judgment, turns out to be the thing AI can’t give you. It can accelerate everything you’ve already built. It can’t build it for you.

Javier put it this way: “The things I’m creating in hours would have taken me a week two years ago.”

One week compressed into hours. But the week didn’t disappear. It compounded. Into taste. Into pattern recognition. Into knowing, before the first line of code is written, whether the architecture is right.

That’s what superpowers actually are. That’s what they’ve always been.

This is the first in a series of conversations I’m having with developers, engineers, and builders across the web data industry. None of them agreed to an interview.

Join our newsletter. If you want to see what AI-powered web scraping looks like in practice, Zyte API is a good place to start.