Zyte Crawls The Deep Web

Explore Memex - The Future of Web Archiving: Learn about Memex, an innovative web archiving system that opens new horizons for preserving digital content.

Shane Evans · Founder, CEO at Zyte

1 min read ·

Zyte Crawls The Deep Web

Zyte crawls the deep web

"The easiest way to think about Memex is: How can I make the unseen seen?"

-- Dan Kaufman, director of the innovation office at DARPA

Zyte is participating in Memex, an ambitious DARPA project that tackles the huge challenge of crawling, indexing, and making sense of areas of the Deep Web, that is, web content not being indexed by traditional search engines such as Google, Bing and others. This content, according to current estimations, dwarfs Google’s total indexed content by a ratio of almost 20 to 1. It includes all sorts of criminal activity that, until Memex became available, had proven to be really hard to track down in a systematic way.

The inventor of Memex, Chris White, appeared on 60 Minutes to explain how it works and how it could revolutionize law enforcement investigations:

new search engine exposes the dark web Lesley Stahl and producer Shachar Bar-On got an early look at Memex on 60 Minutes

Zyte will be participating alongside Cloudera, Elephant Scale and Openindex as part of the Hyperion Gray team. We’re delighted to be able to bring our web scraping expertise and open source projects, such as Scrapy, Splash and Crawl Frontier, to a project that has such a positive impact in the real world.

We hope to share more news regarding Memex and Zyte in the coming months!

Try Zyte API

Build your first scraper in minutes

Free trial, no credit card. From a single request to production in an afternoon.

Get started

Shane Evans

Founder, CEO at Zyte

Shane's articles trace Zyte's history and open-source projects, including the Scrapinghub-to-Zyte rebrand, Portia's visual scraping tool, and Dash. He also wrote on data-storage architecture decisions (why MongoDB was a poor fit for scraped data), remote-work transition lessons f…

More from this author

Continue reading

Konstantin Lopukhin speaking at Extract Summit in Austin
Web data collection

The spider that fixes itself: Help your scraper rise from the ashes

Konstantin Lopukhin explores scrapy-phoenix, an open-source prototype that uses coding agents to repair web scrapers mid-crawl, with validation checks and human escalation.

Konstantin Lopukhin10 min read
Isometric illustration of a website as a portal with an open, glowing door
Access handling

The easiest sites to scrape are the ones that want to be found

Want an easy scrape? Skip the shops and try the military. Defence, government and science sites rank among the most open on the web.

Robert Andrews
Group on stage at Extract Summit Dublin 2026
Web data collection

Five takeaways from Extract Summit Dublin

AI is making scraping code cheaper to write. Extract Summit Dublin explored the harder work that remains: access, infrastructure, maintenance and trust.

Robert Andrews
A line-up of robot babushka or Matryoshka dolls.
Access handling

What web access actually costs - the tier picture

The barrier headlines sound extreme, but the cost data is more differentiated: 87.9% of top sites are Simple or Easy to access - and that means cost-effective.

Robert Andrews
A shopper faces an overwhelming cascade of ecommerce T-shirt product cards in many colours and configurations, illustrating product-variant explosion.
Web data collection

The trouble with variants: Solving the web's infinite ecommerce explosion

Product variants can turn a straightforward ecommerce scraping project into millions of combinations. Learn how to classify, scope and control them.

Robert Andrews12 min read
Two runners carrying backpacks full of web page cards race along a purple track towards a glowing digital finish line.
Web data collection

Need for speed: How data inertia kills business ambition

Slow web data pipelines can hold back new markets, conversion gains and product launches.

James Kehoe8 min read

The Community · Newsletter

The best of Zyte and the data web, in your inbox.

One curated edition — new articles, product updates, and the stories shaping the data web. No noise.