Measure HTML and Markdown Tokens Without Losing Source Content

Measure HTML and Markdown token use on the same source content, then check what was lost. A smaller output may have removed navigation, but it may also have removed the table or qualification needed to answer the question. Token count and evidence retention belong in the same record.

Freeze source content and tokenizer

Interpret the owned fixture honestly

The complete source and candidate below form a small owned fixture with navigation, main text, a table and a code sample. With tiktoken 0.14.0 and cl100k_base, the raw HTML used 94 tokens and the candidate Markdown used 37. The required fixture article content, table and code were retained.

That observation combines markup conversion with removal of boilerplate. It is one fixture, not an estimate of average savings across websites and not a format-only accuracy experiment. Do not extrapolate it to a percentage improvement in RAG accuracy or annual cost.

import tiktoken

html = "<html><body><nav>Home Pricing Login</nav><main>\n<h1>Version 2 extraction fixture</h1><p>The value is 18.</p>\n<table><tr><th>Plan</th><th>Value</th></tr><tr><td>Fixture</td><td>18</td></tr></table>\n<pre>print(18)</pre></main><script>window.fixtureLoaded = true;</script></body></html>"
markdown = "# Version 2 extraction fixture\n\nThe value is 18.\n\n| Plan | Value |\n| --- | --- |\n| Fixture | 18 |\n\n```python\nprint(18)\n```"
encoding = tiktoken.get_encoding("cl100k_base")
assert len(encoding.encode(html)) == 94
assert len(encoding.encode(markdown)) == 37
print(len(encoding.encode(html)), len(encoding.encode(markdown)))

Report a decision your team can reuse

The durable result is a reproducible content-and-token record. The conversion guide explains the request; this page explains how to measure its effect without losing the evidence the application needs.

• Count with the tokenizer used by the target model.
• List missing or altered required evidence before accepting the smaller representation.
• Separate acquisition charges from model token charges.
• For answer-quality claims, use a separate frozen question/evidence evaluation.

Reference documentation: github.com — /openai/tiktoken; www.searchcans.com — /reader-api.

Related implementation and validation guides

• Markdown vs. HTML for LLM Context: Optimizing Performance & Cost