How to convert a website to Markdown for AI: 7 methods compared

Markdown was designed in 2004 so that people could write for the web without writing HTML. Two decades later, it has an unexpected second career: it has become one of the most common ways to feed web content into machines - for model training, retrieval-augmented generation, fine-tuning datasets and AI agents that need to read a page and act on it.

That is why developers now search for things like "convert website to markdown for LLM" and "Markdown for RAG". The question behind those searches is rarely "what is Markdown?" It is far more practical: "How do I reliably produce clean Markdown from pages I do not control, at the scale my pipeline needs?"

This article compares the seven realistic ways to convert a website to Markdown, what each is good at, where each breaks, and how to choose between them - particularly if the output is destined for AI training.

Why Markdown became the default AI input format

Raw HTML is an expensive way to give a model text. Tags, scripts, styles, navigation and repeated site furniture consume tokens, inflate costs and complicate every downstream step. Plain text solves that by stripping everything - including the signals that tell a system how a document is organised.

Markdown sits usefully between the two:

• headings expose the hierarchy of ideas, giving chunkers meaningful boundaries;
• lists and tables keep sequences and row/column relationships intact;
• links preserve references, citations and crawl routes;
• images keep descriptions and source URLs for provenance or multimodal use;
• the result stays human-readable, diffable and easy to audit.

So the format question is largely settled. The hard part is production.

How to think about method comparison

It is tempting to treat "web to Markdown" as a conversion problem. It is not. Converting HTML tags into Markdown syntax is a solved, commodity step - several mature open-source libraries do it well and cheaply.

The real work is a chain of six jobs, and most tools only cover one or two:

1. Discovery and crawl control - finding the right URLs, following canonical links, staying in scope.
2. Access - getting a successful response despite blocks, rate limits, geo differences and anti-bot systems.
3. Rendering - deciding whether the HTTP response is enough, or whether browser rendering, interaction, JavaScript or a specific location is required.
4. Content selection - extracting the page's own content from navigation, cookie banners, related-article rails and ads.
5. Structure-preserving conversion - keeping headings, lists, tables, code, links and reading order while repairing malformed HTML.
6. Dataset preparation - deduplication, provenance, timestamps, language validation, PII handling and refresh policy.

A tool that "converts HTML to Markdown" has solved step five. A pipeline that delivers AI-ready web data has solved all six. Keep that distinction in mind while reading what follows.

Method 1: Use the Markdown the publisher already provides

The cleanest conversion is no conversion at all.

Many documentation sites and static-site generators keep Markdown as their source of truth, and a growing number publish it directly. Three variants matter:

• Direct .md URLs. Some sites expose a Markdown sibling for every HTML page, and sometimes a whole-site Markdown endpoint.
• HTTP content negotiation. Cloudflare's "Markdown for Agents" lets an enabled zone return Markdown from the same URL when the client sends an Accept: text/markdown header, converting the origin HTML at the edge and returning token counts. WordPress plugins increasingly offer the same pattern, sometimes adding .md URLs and rel="alternate" discovery.
• llms.txt. A Markdown file describing a site and pointing agents to useful resources; version 2 also formalises links to page-level Markdown alternatives.

Strengths. Highest possible fidelity to the author's intended structure. No boilerplate classifier needed. Cheap, cacheable and stable.

Limits. Coverage is entirely the publisher's choice, and it is rare across the open web. Markdown variants and frontmatter differ between publishers. Content negotiation only helps on participating sites, and many agent fetchers still never request Markdown. None of it helps if the site blocks you, or if the content you need only appears after interaction. And a publisher offering Markdown says nothing about whether you have the rights to train on it.

Verdict. Always prefer trustworthy first-party Markdown when it exists. Treat llms.txt as a discovery hint. Then assume you still need a universal fallback for the rest of the web.

Method 2: Use an HTML-to-Markdown converter library

These tools take HTML you have already fetched and turn it into Markdown. They are converters, not web-access products.

• Turndown (JavaScript, npm) - a mature, MIT-licensed HTML to Markdown converter for Node and the browser, with rules and plugins for different Markdown styles. Lightweight and easy to embed; it converts whatever DOM you hand it, navigation included.
• markdownify (Python, pip) - an MIT-licensed HTML to Markdown converter built on Beautiful Soup, customisable through options and converter subclasses. A natural fit for Python data pipelines and structurally strong enough for production use.
• Pandoc (command line) - the universal document converter, with an internal document model and broad format support. Excellent for migrations and mixed document types; heavier operationally, and its GPL licensing matters if you intend to link it into a product.
• node-html-markdown, mdream, rehype/remark (JavaScript, npm) - fast, capable whole-page converters. In Zyte's internal testing, a tuned node-html-markdown route that ignored nav and footer achieved about 97% word-level recall, but excluded only 45% of the annotated must-not-survive chrome strings and returned substantially more tokens per page than a main-content approach.

Strengths. Cheap, fast, deterministic, self-hosted and fully controllable. No data leaves your environment.

Limits. They solve only the last mile. Point one at a real web page and it will faithfully convert the cookie banner, the mega-menu and the "you might also like" rail alongside the article. You still own crawling, proxies, browser rendering, main-content selection, retries, monitoring and dataset governance.

Verdict. The right choice when your HTML is already clean and already in hand. An excellent component of a pipeline; not a pipeline.

Method 3: Extract the main content, then convert it

The obvious fix for method two is a cleanup stage first. This two-step design - isolate the content, then convert it - is architecturally sound and widely used.

• Mozilla Readability - the library behind Firefox Reader View, turning a DOM into an article-like object of cleaned HTML and text. Proven and simple, but "readable article" is a narrower target than "keep every useful field". In Zyte's internal testing, Mozilla Readability achieved word-level recall of 0.686, meaning, of the words in the annotated main-content references, its output retained roughly 69% across the benchmark.
• Trafilatura - a capable Python toolkit that combines its own rules with readability and jusText fallbacks, fetches URLs, extracts metadata and can emit Markdown. It produced clean output on text-rich pages, but achieved only about 75% word-level recall in Zyte's testing, discarding materially more words from the annotated main-content references than a recall-led approach.
• Defuddle, jusText and the Boilerpipe family - heuristics based on text density, link density, DOM scoring and stopwords. Inexpensive, explainable and easy to self-host.

Strengths. Far better signal-to-noise than whole-page conversion, at low cost and with full transparency.

Limits. Every extractor encodes a definition of "main content", and most were shaped by news and blog layouts. Product pages, category listings, forums, multilingual pages and unusual templates expose those assumptions - usually by silently dropping content. None of these tools solves access or rendering.

Verdict. Strong for homogeneous, article-shaped corpora. Riskier the more varied your page types are.

Method 4: Build on a crawler framework

If you want the whole pipeline in-house, frameworks give you the scaffolding.

Crawl4AI is an open-source browser and crawler framework that produces raw Markdown and can add pruning or BM25-based "fit Markdown", with support for rendered pages, sessions, actions and configurable filters. In Zyte's internal testing, a tuned pruning route achieved about 90% word-level recall, but also retained substantially more clutter and emitted many more tokens per page than a main-content route. Note also that BM25 "fit" output is query-specific - useful for one RAG question, less useful as a neutral, reusable corpus.

The fully custom stack is the other version of this method: Playwright or Selenium, proxies, Beautiful Soup or Cheerio, Readability or Trafilatura, and Turndown or markdownify.

Strengths. Maximum control, no mandatory data egress, and the ability to tune hard for a specific domain.

Limits. The cost does not disappear, it relocates - into proxy and access operations, browser upgrades, selector drift, retry logic, observability and continuous quality evaluation. That is a standing engineering commitment, not a one-off build.

Verdict. Justified for high-value, bounded domains where custom logic genuinely beats general-purpose extraction. Expensive as a general solution.

Method 5: Call a hosted URL-to-Markdown API

This category collapses fetching, cleaning and converting into a single request. It is where most teams land, and where the quality differences are largest.

• Firecrawl exposes scrape, crawl, search and document parsing, with Markdown as a default output, main-content extraction on by default, and PDF and Office support. In Zyte's internal testing on the same 299 pages, it retained the same share of main content as Zyte's route (95%), but removed far less boilerplate (61% versus 85%) and returned 57% more tokens.
• Jina Reader converts a URL or supplied HTML into LLM-friendly Markdown, with selector, caching, image and ReaderLM options, and an exceptionally simple URL-prefix interface. In Zyte's testing, its default behaved like a whole-page reader: it achieved very high word-level recall, but also retained a large amount of page furniture. Explicitly stripping header, footer, navigation and sidebars improved cleanliness on a smaller probe. Its ReaderLM-v2 option was much slower and sharply reduced word-level recall as well as link and table-cell retention.
• Context.dev offers URL-to-Markdown, crawling, browser actions, automatic proxy escalation, broad file support and a useMainContentOnly mode. On the same 299-page benchmark, its main-content mode retained slightly more main content than Zyte's route (97% versus 95%), but removed less boilerplate (71% versus 85%) and returned 39% more tokens. It retained more content on product pages, but Zyte's route achieved the stronger balance of recall and precision across every page type.

Strengths. Fastest time to value. Crawl controls, caching, document handling and output options are mature across this category.

Limits. Recurring cost, data egress, cache-freshness defaults, closed extraction logic and complete dependence on someone else's definition of main content. And crucially: a clean converter is worthless without reliable access to the page in the first place.

Verdict. The pragmatic default - but benchmark candidates on your page types rather than on article samples, and check access success separately from extraction quality.

Method 6: Let a model do the conversion

A newer approach uses AI to classify or extract web content, rather than relying on fixed DOM rules, and can generate cleaned Markdown from the selected blocks. ReaderLM, MinerU-HTML and Pulpie are representative.

Strengths. Can infer structure when HTML is irregular or semantically meaningless, and can merge extraction and conversion into one step. Output is often strikingly clean and compact.

Limits. Slower and more expensive than deterministic conversion. Non-determinism and truncation are awkward when you need reproducible datasets. Generated text needs checking for omission and fabrication. Licensing is a real obstacle: in Zyte's testing, MinerU-HTML emitted no links at all and carried a revenue-capped licence, and ReaderLM-v2's weights were non-commercial.

The headline results deserve care. Pulpie Orange large and MinerU-HTML both produced cleaner-looking output than Zyte's route, but their word-level recall was only about 82-84%, compared with about 95% for Zyte's recall-led approach. Cleaner is not the same as better.

Verdict. Promising as a selective fallback for messy document types. For bulk training corpora, determinism, field fidelity, speed and licence clarity usually matter more than concision.

Method 7: Get access and extraction from one API - Zyte API's Markdown output

Zyte API's new Markdown output addresses the two problems the other methods leave open: reaching the page, and choosing the right part of it.

Set "markdown": true and the API returns a plain Markdown string. Two scopes are available: mainContent, where the same detection machinery behind pageContent identifies the primary HTML region, which is then repaired and converted; and fullPage, which converts the whole document. You can work from httpResponseBody or browserHtml, and you can request Markdown alongside structured pageContent fields from a single analysis pass.

Four things distinguish it:

1. Access is part of the product. Anti-ban handling, proxy management and browser rendering are not a separate layer you assemble and operate. On difficult sites, this is usually the difference between a dataset and an error log.
2. Extraction is recall-led. In Zyte's internal testing, the method retained about 95% of main content, removed about 85% of unwanted page chrome and cut the token count by roughly 97% compared with source HTML.
3. Structure survives because we convert HTML, not text. Zyte does not flatten the page and reconstruct pseudo-Markdown. The main-content model identifies the relevant HTML subtree, and that subtree is converted - so headings, lists, tables and links come through.
4. Navigation can stay structured. Request markdown and pageContent together and your agent reads a compact body document while your application handles breadcrumbs, pagination and next-page links as explicit fields rather than as prose it has to parse.

The seven methods, side by side

If the output is for AI training, optimise for recall first

Across these methods, the central trade-off is easy to read the wrong way: the cleanest-looking output may have got there by discarding more of the page.

For training data and general-purpose fetching, that is the wrong trade. Leaked boilerplate is recoverable: you can filter it, deduplicate it or down-weight it later, and its main cost is tokens. A missing paragraph, price or table row is gone, and no downstream step can reconstruct it. Recall protects options; cleanliness spends them.

Narrow RAG use cases can justify aggressive query-focused pruning - but the resulting document is tuned to one question and is no longer a neutral, reusable source.

What this means for RAG and agent pipelines

If the destination is a retrieval index rather than a training run, the Markdown you produce has to survive three more stages, and extraction quality shows up in all of them.

• Chunking. RAG pipelines often use header-aware splitters - LangChain's MarkdownHeaderTextSplitter and its equivalents in LlamaIndex and Haystack - which split on # levels and attach the heading path as metadata. That only works if headings survived extraction. Whole-page output breaks it differently: navigation lists and footer links become their own chunks, and they retrieve well against short queries because they are dense with keywords.
• Document processing alongside the web. Most production AI datasets mix web pages with PDFs, Office files and docs sites. Normalising everything to Markdown first gives one downstream path, so check that whatever you pick handles non-HTML input or that you have a second tool for it.
• Agent fetch and MCP. When a coding or research agent reads the web through an MCP server, its fetch tool is your extraction layer, and its output lands directly in a context window you are paying for. The same main-content logic behind Zyte API's Markdown output powers Zyte's MCP fetch tool, which is why token efficiency and recall matter as much here as in a batch pipeline.

Markdown is a milestone, not the finish line

Whatever method you choose, valid Markdown is not the same as a training-ready corpus. A perfectly formatted document can still carry duplicated templates, cookie text, broken reading order, tracking URLs, hidden prompt injection, personal data or content you have no right to use.

Before a corpus is genuinely usable, you also need:

• a source and licence policy, with canonical URLs and capture timestamps recorded;
• deduplication at page, paragraph and template level;
• language and encoding validation;
• preservation tests for the fields that matter most;
• removal or quarantine of PII, secrets and unsafe material;
• stable normalisation rules and versioned extractor configuration;
• contamination checks between training and evaluation sets;
• a defined policy for updates and deletions.

No extraction API removes that work. But it can stop adding to it.

How to choose

1. Take first-party Markdown where a publisher offers it and you trust the source.
2. Otherwise, start with access. Extraction quality is irrelevant on pages you cannot fetch or render.
3. Default to high-recall main-content extraction rather than whichever tool produces the tidiest output.
4. Keep a full-page fallback and structured navigation for the cases where surrounding regions matter.
5. Benchmark on your own page types - especially commerce and listing pages, where article-tuned tools quietly fail.
6. Validate the result as a dataset, not merely as Markdown.

If that sequence describes your problem, Zyte API now handles the middle of it in a single request. Set "markdown": true and see what comes back.

Frequently asked questions

Is Markdown better than HTML for LLMs?

For most reading and retrieval tasks, yes. Markdown keeps the structural signals a model can use - headings, lists, tables, links - while discarding the tags, scripts, styles and navigation that dominate an HTML page's token count. In Zyte's internal testing, main-content Markdown used roughly 97% fewer tokens than the source HTML. The exception is when you specifically need attributes, embedded data or the layout itself, in which case raw HTML is the right input.

What is the best HTML to Markdown converter for Python?

markdownify is the most common choice for pipeline work: MIT-licensed, built on Beautiful Soup and customisable through converter subclasses. Trafilatura is the better pick if you also want main-content extraction and metadata in one step, and Pandoc is worth it when you are converting mixed document formats rather than web pages. Remember that all three convert the HTML you give them - none of them fetches or unblocks pages.

How do I convert a whole website to Markdown, not just one page?

You need crawl management in front of the converter. Either build a crawler on a framework such as Crawl4AI, or use an API with crawl endpoints - Firecrawl and Context.dev can discover and fetch pages at scale, while Zyte API handles page access and extraction. Check whether a site publishes llms.txt or site-wide Markdown endpoints first, because that can save the crawl entirely. At scale the bottleneck is almost always access and crawl management, not conversion.

Can I just use a Chrome extension or an online converter?

For a handful of pages, yes - extensions such as MarkDownload do the job and keep processing local. They are the wrong tool for anything repeatable: no crawl coverage, no provenance, inconsistent output between runs and no way to refresh thousands of pages on a schedule.

What does "main content" extraction actually remove?

Main-content extraction removes site-wide furniture: header and footer navigation, cookie and consent banners, sidebars, ad slots, social buttons, related-article rails and repeated template text. What counts as furniture is a judgement call encoded in each tool, which is why extractors differ so much on the same page - and why article-tuned tools often strip genuine content from product and listing pages.

Is Markdown extracted from the web ready to train on?

Not on its own. Clean Markdown removes the formatting problem, not the corpus problem: a production-ready AI dataset still needs licence and rights review, canonical URLs and capture timestamps, deduplication, language validation, PII handling, and contamination checks between training and evaluation sets. Good extraction stops you adding to that work; it does not replace it.