The trouble with variants: Solving the web's infinite ecommerce explosion

When is a t-shirt not a t-shirt? When it’s 3.5 million possible permutations of that same t-shirt.

That is what happens when a customisable garment offers two base designs, 21 body colours, 21 zipper colours, 19 chest-logo options, 19 pouch-logo options, and 11 sizes. To a shopper, it is just one humble item of clothing. To a web scraping project, it can be millions of combinations — and the mother of all headaches.

Product variants rarely arrive at a data engineer’s door framed as a multimillion-combination problem. The scoping call sounded straightforward: the customer just wants product data from 40 retail sites — title, price, availability, images; the usual. Then, in the last minute, almost as an afterthought: “Oh, and we’ll want variants, obviously.”

That one word can double the build, triple the quality assurance (QA), and rewrite the commercial terms of the project. It is probably the single most under-discussed source of trouble in ecommerce web scraping, and it almost always arrives late, after the crawlers are written and the first sample has been signed off.

The reason is simple — everyone in the room assumes “variant” means the same thing. It does not.

Variants are in the eye of the beholder

Imagine a jacket listed on a retailer site. Is that one product?

That depends on who you ask.

All of those are true at the same time — and that’s not necessarily helpful.

How products multiply

Variant multiplication leads a web crawl down a path to what mathematicians call a “combinatorial explosion,” the web scraper’s nightmare equivalent of Inception.

That single t-shirt can easily become 3.5 million combinations. Multiply again by the three base product types on the same page and the total approaches 10 million. Don’t look now, but your target website has 170 customisable products. Good luck with that project.

Imagine a configurable smartwatch with four component choices: 18 bands, seven clasps, seven hooks, seven links. That is 6,174 combinations for a single listing.

David Comber, Solutions Architecture Manager at Zyte Data, Zyte’s done-for-you expert data extraction service, warns:

“Treating every variant as a separate product can simplify the data structure, but it changes the number of records — and that changes the pricing and the estimate."

Variants are scrapers’ scars

A data engineer will tell you about their most taxing dataset haul the way a fisherman tells you about the 900lb whopper that got away.

On one large ecommerce marketplace tackled by the Zyte Data team, more than half the products crawled had variants — 13.3 million of them across 402,000 products. One listing had a humongous 4,900 permutations.

Because price and availability were only exposed for each of those variants, each one needed its own request. That worked out at 38 requests per product, an estimated 210 million requests, and a crawl that would have taken the best part of 10 months to finish.

None of those projects were badly built. No, they ran into hard arithmetic. That has knock-on effects that have nothing to do with crawling per se:

• Single product records growing past the object size limits of a delivery API, forcing a switch to bucket delivery.
• Synchronous extraction timing out because one product cannot be resolved in a single call.
• Monthly delivery volumes that no longer match anything in the original estimate.

Says Sigit Dewanto, Team Lead, Zyte Data:

“We’ve seen a single product with up to 10 million variants. When the customer saw what collecting that would involve, they decided not to scrape the product at all.”

Zyte Data’s four varieties of variants

The trouble is, target sites (and their ecommerce platforms) are heterogenous — there is no consensus on their part either about what a “variant” truly is.

“There isn’t one way to handle variants. Every website structures them differently, so you need rules for each site,” Comber adds.

This is the part that makes a universal rule impossible, and it is where the Zyte Data team's own classification framework earns its keep.

In the framework used by Zyte Data’s data engineers, there are four ways products present themselves online, and each one demands different crawling logic:

1. Standalone

A unique product with no variations visible to the buyer. There’s nothing to decide. These are the ones that make a project look easy in the sample.

2. Parent

One item on the shelf, with options selectable inside the detail page. Selecting an option typically does not produce a new URL, and the options do not appear separately on category or search pages.

Sites like this usually carry a stable identifier for the parent and a second identifier for the selected option, with one option pre-selected on arrival. Get this wrong and you collect one row per product where the customer expected 11.

3. Variant (exploded)

The same product, “exploded” onto shelf pages as independent links. Each option is its own listing, they share most of their attributes, and they are cross-linked through swatches or similar.

There is often no identifiable parent at all, a ghost parent, though frequently there is a product group identifier buried in the page that ties the family together. Collect these naively and you get duplicates, inconsistent grouping, and an item count nobody can reconcile.

4. Bundle

A page that combines the previous three types into one buying unit — for example, a furniture suite made of a standalone sofa plus a parent item with a pre-selected option.

According to Akhter Wahab, Team Lead, Zyte Data:

“A bundle is not another kind of variant. It is a different entity and should be classified separately.”

Treating a bundle as a parent produces records that are wrong in a way that is very hard to spot downstream.

A single retailer can use more than one of these patterns across its catalog. Two retailers in the same vertical will rarely use the same one.

How Zyte API sees product variants

Zyte API, Zyte’s self-service extraction tool, can see, understand and extract product variants as part of its AI-powered automatic extraction of ecommerce product listings.

This helps data gatherers identify and extract key product data from an item, as well as its sibling items.

1{
2  "product": {
3    "variants": [
4      {
5        "name": "Navy",
6        "url": "https://www.example.com/product/sku-123?color=navy",
7        "sku": "SKU-123-NAVY",
8        "price": "39.99",
9        "currency": "USD",
10        "availability": "InStock",
11        "selected": true,
12        "properties": {
13          "color": "Navy",
14          "size": "M"
15        },
16        "images": [
17          {
18            "url": "https://www.example.com/images/sku-123-navy-front.jpg"
19          },
20          {
21            "url": "https://www.example.com/images/sku-123-navy-back.jpg"
22          }
23        ]
24      }
25    ]
26  }
27}

To Zyte API, a “variant” is:

• a version of a base product that differs by one or more defined attributes (most often, colour, size, material, or configuration).
• displayed or made viewable from the base product page.

For the API, products that look like obvious siblings to a human but are not actually linked to each other from the base page are not variants; they are merely products that happen to resemble each other.

That definition is deliberately narrow for a reason. Without a boundary, your brief to “get all the variants” becomes an instruction to infer a retailer's entire internal product hierarchy from the outside.

How deep do you go?

Once you know which pattern a site uses, the next question is how far into it you are willing to reach. There are three practical levels, and they differ by an order of magnitude in cost.

The default position, and the one that keeps projects affordable, is to collect what is visible on the shelf and the detail page, and to avoid additional requests per variant unless the customer's use case genuinely needs them.

Zyte API’s automatic extraction works on what is present on the page it is given. Variant data that only exists behind a click, a chained request, or a per-option API call is a different job.

There is a trap here, too. Option sets are often conditional: choosing a colour changes which sizes are offered. But the valid combinations only appear one branch at a time, which means selecting an option, waiting for the page to update, reading the next set, and repeating. Visible options and valid purchasable combinations are two different deliverables, and the gap between them is where QA iterations go to die.

Identity is the other half of the problem

Assume you have collected everything correctly. You still have to say what each record is.

SKUs and barcodes are not reliably sufficient. Some sites reuse them, some omit them, some expose one identifier on the shelf and another on the detail page. Carrying an explicit parent identifier in the schema is the difference between a dataset you can group and a dataset you can only count.

It matters commercially as well. Flattening variants into individual products simplifies the structure and is often the pragmatic choice on high-variant sites, but it changes the record count, and record count is usually what the contract is priced on.

Deduplication is the sharp edge. On one kit-style product, variant identifiers matched the identifiers of genuinely independent listings. A duplicate detector dropped those listings, even though they had their own URLs and their own prices and should have been collected separately. The moral of the story is — a variant and an independent product can be the same object wearing two hats.

How to scope variant extraction properly

Want to avoid the blast zone of the “combinatorial explosion”? Here are Zyte Data’s hard-won tips worth stealing for your next brief:

1. Start with the business question, not the field. “Are we price-competitive on every size we sell?” leads to a different project than “which competitors stock this range?” The question determines the depth, and therefore the cost. Don’t make “all variants” a requirement, make it a placeholder.
2. Classify each target site. Standalone, parent, exploded variant, or bundle? Do this before estimating anything. A 40-site project is usually a mix, and the mix is the estimate.
3. Choose a depth per site, not per project. Basic for the long tail, enhanced for the handful of sites where per-option price and stock actually drive a decision.
4. Decide the record shape up front. Nested variant arrays under a parent, or one row per variant. Decide which identifier makes a variant a separate record, and what happens when a variant also exists as a standalone listing.
5. Agree a stop rule for explosion. Before a crawl meets a product with a million combinations, decide what happens: limit to base choices, collect components rather than combinations, cap the variant count, or exclude the product. Also agree that if a non-base choice turns out to change price or availability, that gets reported and revisited.
6. Hand over hard examples, not easy ones. Five to 10 URLs, deliberately including a conditional option set, a variant with its own URL, a variant with a distinct identifier, a bundle, and something out of stock. Along with the output you expect for each.
7. Expect per-site rules and budget QA for them. Variant work adds review cycles because completeness is difficult to prove: a missing branch looks exactly like a product that simply had fewer options. Teams that have done this repeatedly plan for several QA passes per crawler on complex variant sites, and roughly a day or more of extra development per site.