When data-hungry companies talk to Zyte Data, our done-for-you web data gathering service, about acquiring web data, they are seldom choosing between two providers.
More often than not, the comparison is this: Zyte Data, or an in-house build?
Scratch beneath that question, however, and you’ll find the comparison is not actually a simple run-off between outsourcing and internal investment. “Build or buy?” is about more than just vendor selection and more than assignee capability. The real test is about resources.
When an organisation already has a data engineering team, even a functioning data pipeline, the real question it has to ask itself is: “Can we get the outcome we need at the pace we require?”
That’s why, when it comes to managed web data gathering services - and the businesses that depend upon them - speed is becoming the critical determinant.
In-house inertia
There are some very smart data engineers out there. Most teams can build a rudimentary data pipeline; they can get a first version working. All too often, however, they discover that the work simply doesn’t end there.
Initial estimates are often surface-level deep: target site selection, extraction logic and breaking ground on the first few records. Here’s what happens next:
Coverage grows quickly. The first sites are usually the simple, well-structured ones. The valuable sources tend to be less consistent and more difficult to extract. A target list of 20 sites can become 300, and the last 200 can take longer than the first 100. The hidden costs of running your own pipeline are mostly found in that long tail.
Targets change and protections change with them. An extractor that works today can fail after a site changes its layout or deploys new bot controls. Fixing that is ongoing engineering work, not a one-time project. It is also a common reason an in-house pipeline gradually loses coverage. Anti-bot detection has changed substantially, and keeping up is part of the job.
A schema becomes a promise. Once a downstream product or process relies on a field, changing that field is no longer a quick edit. It may require coordination, backfills and release work. Teams often learn this only after the first consumer has gone live.
Maintenance takes over. Building the pipeline is the visible milestone. Keeping it running while hundreds of sites change underneath it is the permanent workload.
These snags are not a reflection on the in-house team. Many experienced engineering groups run scraping at scale. The issue is that web data infrastructure is larger than it first appears, and it continues to compete with the rest of the roadmap.
The business cost of slow data
It’s never about the data alone. For organisations, that is always a means to an end - a raw resource to power a new business initiative.
A clog in the pipes that get data flowing usually affects one of three commercial goals.
1. Opening a new market
A new geography, category or segment may depend on coverage you do not yet have. Until the data is ready, that market stays closed.
This is expensive precisely because it is easy to miss. Revenue you never booked does not appear in the accounts as a loss. To estimate the cost, take the annual revenue you expect from the market, divide it by 12 and multiply it by the additional months required to build rather than buy.
If a region is expected to generate £1 million a year and an internal build adds nine months, roughly £750,000 of potential revenue arrives later than it could have. If the decision itself is deferred, the same problem rolls forward.
2. Improving conversion
The data may help you build a more useful product, differentiate an offer or improve a commercial motion. Until it is available, though, customers see the version of the offer you already have - while competitors release new, data-rich features.
Conversion gains apply to the volume already moving through the business. Each month of delay is a month when that lift is absent from the existing base. Even a modest improvement can become material when applied across current revenue.
3. Shipping velocity
Data work often becomes a queue behind a finite number of people. Launch dates start to depend on when the data team has capacity, and one team’s availability begins to shape the whole roadmap.
The cost is not limited to a single delayed launch. If data engineering constrains how much the company can ship in a quarter, removing that constraint can change the delivery rate over time. One Zyte Data customer described the commercial stake like this:
“Sales can double revenue if we half delivery time.”
In each case, the build cost is only part of the decision. The revenue delayed by the build is often the larger number.
A framework for data in-housing
There are cases where building in-house is the sensible choice.
| Build in-house when… | Why |
|---|---|
| Extraction method is your differentiator | The pipeline itself is core product IP. |
| Targets are few and stable | The source set is well-structured and manageable internally. |
| A suitable API exists | Custom extraction may be unnecessary. |
| The need is one-off | A standing service is hard to justify. |
| No commercial deadline depends on it | The cost of waiting is low. |
| You must own the pipeline | Contractual or regulatory needs outweigh speed. |
If several of these conditions apply, build it in-house. A managed service may not be the right fit, and we would be the first to say it.
Put a number on the wait
If time-to-data is a central factor in allocating your data pipeline budget, then model timing and compare potential providers. To do so, you need four inputs:
- Annual revenue the data could unlock once it is fully in use.
- Duration and cost of an internal build, including maintenance rather than only the first release.
- Time to usable data and annual cost of a partner.
- Ramp period and the comparison horizon.
The model should show the revenue captured by starting earlier, the number of months gained and the net difference after costs.
Use ranges where the evidence is uncertain. Treat revenue as an upside case rather than a promise, and treat the internal build estimate as a lower bound rather than a fixed date. A conservative midpoint, supported by a range, is more useful in a finance review than a single precise-looking forecast.
What fast can look like
Where the scope is clear and the decision has been made, at Zyte Data we have taken teams from a first conversation to a live, billable production feed in a week. Every feed in scope was delivered in about a month.
Here are some recent customer executions that stick in my mind:
| Company/project | What was delivered | Time | Context |
|---|---|---|---|
| Multi-feed data project — first milestone | A live, billable data feed. | Less than one week from the first conversation. | Requirements were clear and the customer was ready to proceed. |
| Enterprise commissioning retailer feeds | Eight of the first ten complete feeds. | Two weeks. | Four feeds per week. |
| Global ecommerce marketplace | 83 of 112 retailer product feeds ready for review or approved. | About three weeks. | Development continued while the customer reviewed completed batches. |
| Multi-feed project — full rollout | Every feed in the agreed scope. | One week to one month. | The first production feed went live inside the first week. |
One customer once described the standard well to us: “As quick as you can work on this, we can work.” That is a reasonable standard for any provider.
And we are quite proud to say we meet that standard. Here’s what one retail distribution company told us earlier this year:
“We really appreciate that the solution was developed and deployed in such a short period of time… Outstanding is an understatement.”
That’s why Zyte Data customers are leaving reviews like these on enterprise review sites:
“I loved the ease of the initial setup with Zyte, as they took care of all the development. We only needed to communicate what data we needed and set up the necessary processes on our end. I find it great that Zyte handles everything for us, allowing us not to worry about maintaining a platform ourselves.”
The next step
If a commercial goal is waiting on data you do not yet have, before reopening the standard “build-versus-buy” playbook, you need to quantify the cost of delay.
Our supplier return-on-investment calculator uses your inputs to model a multi-year view, helping you understand:
- Revenue you could capture sooner.
- Months gained.
- Net position after costs.
It presents each as a range rather than one optimistic figure.
If your question is about the cost of extracting individual sites, we have a separate estimator for that.
Do it now - time is of the essence.