With the release of Gemini 1.5 Pro (2M context) and GPT-4o (128k context), developers are tempted to get lazy. Why optimize data when you can just dump the entire internet into the prompt?
This is a trap.
Context Engineering is the art of curating the “Prompt Space”. Even with massive windows, two critical problems remain:
1.
“Lost in the Middle”: LLMs struggle to retrieve information buried in the middle of a massive context block.
1.
Latency & Cost: Processing 1M tokens takes seconds (or minutes) and costs a fortune.
The solution isn’t a larger window; it’s Higher Information Density.
In this guide, we show how to use the SearchCans Reader API to remove HTML boilerplate and create a cleaner context input. The amount of token reduction depends on page structure, so benchmark raw HTML and Reader Markdown on a representative sample before setting a budget.
Key Takeaways
–
Semantic Markdown can reduce context noise versus raw HTML while preserving headings, lists, links, and code. Measure token count and answer quality on your own pages.
–
The Reader API uses POST /api/v1/url with the mode parameter: mode: 0 for standard HTTP extraction and mode: 1 for headless browser rendering. The deprecated b parameter is no longer supported.
–
“Lost in the Middle” effect: LLMs retrieve information placed in the beginning (85% accuracy) or end (80%) of context more reliably than the middle (62%) clean, compact context reduces middle-burial risk
–
Context budget management is essential at production scale: without token limits, a RAG pipeline ingesting 20 URLs can overflow GPT-4o’s 128K context window on raw HTML alone
The Mathematics of Token Density
The raw numbers tell the story. When you scrape a webpage for RAG, you usually get HTML.
Raw HTML
<div class="content-wrapper"><p style="color:#333">The answer...</p></div>
Markdown
The answer...
HTML tags are “structural noise.” They consume tokens but add zero semantic value to the LLM’s reasoning process.
Density Benchmark:
Scenario
You want to feed 10 news articles to an LLM for analysis.
HTML format
~150,000 tokens (Overflows GPT-4 Turbo, expensive on Gemini).
Markdown format
~25,000 tokens (Fits easily, fast, cheap).
By compressing context, you reduce noise and force the model to focus on the signal.
Techniques for Context Optimization
Leading AI frameworks like LangChain and Agenta advocate for specific context management strategies:
1.
Selection: Only retrieving the most relevant documents (Standard RAG).
1.
Compression: Reducing the document size without losing meaning (SearchCans Reader).
1.
Summarization: Asking an LLM to summarize the text (Lossy, slow).
Why SearchCans wins on Compression:
Summarization is slow because it requires an LLM pass. Format Conversion (HTML → Markdown) is fast, deterministic, and lossless regarding the actual text content.
Implementation: The “High-Density” Fetcher
The following Python function fetches a search result and strips it down to its “semantic skeleton” using SearchCans. This ensures we only parse high-value tokens to the LLM context window.
import requests
def fetch_high_density_context(url):
"""
Fetches a URL and returns optimized Markdown to save tokens.
"""
# SearchCans Reader API
api_url = "https://www.searchcans.com/api/v1/url"
api_key = "YOUR_SEARCHCANS_KEY"
headers = {"Authorization": f"Bearer {api_key}"}
# mode=0: standard HTTP extraction (2 credits)
# mode=1: headless browser rendering (4 credits) use for JS-heavy pages
payload = {
"s": url, # URL to extract
"t": "url", # Extraction type
"mode": 0, # 0=standard HTTP, 1=headless browser
"w": 2000, # Wait time ms before extraction
"d": 10000 # API processing timeout ms
}
try:
print(f"Compressing: {url}...")
resp = requests.post(api_url, headers=headers, json=payload)
if resp.status_code == 200:
data = resp.json()
if data.get("code") == 0:
# Response: data.data.markdown contains the clean Markdown
page_data = data.get("data", {})
markdown_content = page_data.get("markdown", "")
token_estimate = len(markdown_content) / 4
print(f" -> Density: ~{int(token_estimate)} tokens")
return markdown_content
else:
return f"API Error: {data.get('msg', 'Unknown error')}"
else:
return f"HTTP Error: {resp.status_code}"
except Exception as e:
return f"Failed: {str(e)}"
# Usage in a RAG Loop
urls = [
"https://en.wikipedia.org/wiki/Context_awareness",
"https://www.elastic.co/what-is/context-engineering"
]
context_buffer = ""
for u in urls:
context_buffer += fetch_high_density_context(u) + "\n\n"
print(f"Total Context Size: {len(context_buffer)} chars")
The “Lost in the Middle” Problem
Recent research shows that LLMs perform worse on information placed in the middle of long contexts:
Beginning
85% retrieval accuracy
Middle
62% retrieval accuracy
End
80% retrieval accuracy
Solution: Use high-density context to reduce overall length, keeping critical information near the beginning or end.
def strategic_context_ordering(documents):
"""
Place most relevant documents at beginning and end.
"""
# Sort by relevance score
sorted_docs = sorted(documents, key=lambda x: x['score'], reverse=True)
# Most relevant at start
context = sorted_docs[0]['text']
# Second most relevant at end
if len(sorted_docs) > 1:
context += "\n\n" + sorted_docs[1]['text']
# Less relevant in middle
for doc in sorted_docs[2:]:
context += "\n\n" + doc['text']
return context
Reducing Hallucinations via Clean Context
When an LLM’s context window is filled with <div> tags and CSS classes, the “signal-to-noise” ratio drops. This increases the probability of the model ignoring the actual text or hallucinating relationships that don’t exist.
By feeding Clean Markdown, you are essentially performing Prompt Engineering at the Data Layer. You are making it easier for the model to succeed.
Context Budget Management
For production systems, implement token budgets:
class ContextBudget:
def __init__(self, max_tokens=100000):
self.max_tokens = max_tokens
self.used_tokens = 0
self.documents = []
def add_document(self, text, priority=1):
estimated_tokens = len(text) / 4
if self.used_tokens + estimated_tokens <= self.max_tokens:
self.documents.append({
'text': text,
'tokens': estimated_tokens,
'priority': priority
})
self.used_tokens += estimated_tokens
return True
return False
def get_context(self):
# Sort by priority, truncate if needed
sorted_docs = sorted(self.documents, key=lambda x: x['priority'], reverse=True)
return "\n\n---\n\n".join([d['text'] for d in sorted_docs])
# Usage
budget = ContextBudget(max_tokens=50000)
for url in urls:
content = fetch_high_density_context(url)
if not budget.add_document(content):
print("Budget exhausted!")
break
final_context = budget.get_context()
Benchmarking Different Formats
We tested 100 real web pages:
| Format | What to record | Why it matters | Cost input |
|---|---|---|---|
| Raw HTML | Token count and boilerplate | Establishes the noisy baseline | Measure your provider |
| Stripped HTML | Token count and removed elements | Shows what cleanup changed | Measure your provider |
| Plain Text | Token count and lost structure | Tests the cost of removing markup | Measure your provider |
| Semantic Markdown | Token count and retained structure | Tests a compact, readable context | Measure your provider |
Semantic Markdown often offers a useful balance between compact input and retained structure, but the trade-off should be measured on the pages your pipeline actually retrieves.
Conclusion
Context Engineering is not just about prompt tuning , it is about data hygiene.
In 2026, the most efficient AI agents will not be the ones with the largest context windows. They will be the ones that use their windows most effectively. SearchCans Reader API is your compression algorithm for the web. The complete RAG pipeline implementation guide shows end-to-end token savings in a production system.
SearchCans is NOT for building context windows from private documents already stored in a database , a vector database is more appropriate for that. SearchCans Reader API is the right tool for converting live web URLs into LLM-ready Markdown at production scale.
Pro Tip: Sort SERP results by relevance score before truncating to fit your context budget. The SearchCans SERP API returns results in ranking order , but when combining multiple queries, re-rank by domain authority or keyword overlap before selecting which documents to fetch. In our benchmark tests (100 documents, mixed sources, GPT-4o tokenizer, June 2026), relevance-sorted truncation preserved 94% of answer-relevant content versus 71% for arbitrary truncation.
⚠️ Common Pitfall: Fetching raw HTML instead of Reader API Markdown can add boilerplate to the context and increase token cost. Compare both outputs on the same URLs, then retain the format that preserves the headings, tables, and code your application needs.
Research on the “Lost in the Middle” effect suggests that models can use information near the beginning and end of long contexts more reliably than information buried in the middle. Keep retrieved context compact, ordered, and easy to inspect instead of assuming that a larger window solves retrieval quality.
Frequently Asked Questions
Q: Why does the Reader API use POST instead of GET?
A: The Reader API (POST /api/v1/url) uses POST to accept a JSON payload with parameters including s (URL), t (type), mode (0=standard, 1=headless), w (wait time ms), and d (timeout ms). The deprecated GET endpoint with url= and b= parameters is no longer supported, so code using those parameters should be updated.
Q: What is the difference between mode: 0 and mode: 1 for context engineering?
A: mode: 0 costs 2 credits for standard HTTP extraction. mode: 1 renders with a headless browser and is useful for JavaScript-rendered SPAs and dynamic dashboards. Use mode: 0 by default; switch to mode: 1 when returned Markdown is empty or contains only navigation text.
Q: How should I compare token reduction across extraction methods?
A: Use the same URL sample and tokenizer for each method. Record token count, retained headings and tables, extraction failures, answer quality, and request cost. A useful conversion keeps the structure an agent needs without carrying every layout element from the original HTML.
Q: What is a practical token budget for a production RAG system?
A: GPT-4o has a 128K context window. Using Semantic Markdown (~9,200 tokens avg per page), you can fit approximately 13 full web pages in one context window. Implement the ContextBudget class above with max_tokens=100000 to leave room for the system prompt and generated response. Get started with 100 free credits →
