Missing Link in RAG: Search-to-Markdown Pipeline (Python)

Key Takeaways

• Two-step pipeline: SearchCans SERP API (POST /api/search) finds URLs → Reader API (POST /api/url) converts top results to clean Markdown — both use the same API key and pay-as-you-go wallet
• mode: 1 not b: True — headless browser mode requires "mode": 1 in the Reader API payload. The deprecated b parameter is silently ignored, causing dynamic pages to return empty content
• Use json= not data=json.dumps() — pass dicts directly to requests.post(json=payload). Using data=json.dumps() requires correct Content-Type header and is error-prone
• Parallel fetching cuts latency by 5×: reading 5 search results sequentially takes ~15s; running them concurrently with concurrent.futures.ThreadPoolExecutor takes ~3s

Introduction

90% of "RAG" (Retrieval-Augmented Generation) tutorials on the internet are fundamentally broken. They teach you how to chat with a static PDF you uploaded three weeks ago.

But in 2026, useful AI Agents need live data. If your agent cannot read today’s news or check a competitor’s pricing page right now, it is hallucinating on obsolete information.

The Solution? You need a pipeline that connects Real-Time Search with Clean Markdown Extraction.

In this guide, we will bridge the gap between "searching" and "reading." We will build a Python pipeline that searches Google, extracts the most relevant URLs, and converts them into clean, LLM-ready Markdown—all in under 3 seconds.

Why Raw HTML Kills RAG Performance

You might be tempted to just use a simple HTTP request to dump HTML into your vector database. This is a critical mistake that leads to "poisoned" context.

The Token Waste Nightmare

Raw HTML is notoriously inefficient. A typical news article might contain 50KB of text but 2MB of HTML tags, scripts, and CSS.

The Cost Factor

You are paying for tokens that convey no meaning (<div>, class="nav-wrapper").

The Impact on Context

You fill up the LLM’s context window with garbage, pushing out the actual relevant data.

Semantic Noise & Hallucinations

Vector databases embed everything you feed them. If you feed raw HTML, your RAG system might retrieve a footer link about "Careers" instead of the product pricing you asked for.

Pro Tip: The "Nav-Bar" Trap

Standard scrapers often capture the navigation bar on every single page. This tricks the embedding model into thinking every page on a website is about "Home / About / Contact," diluting your search relevance.

The Architecture: Search-to-Markdown

To fix this, we need a "Golden Duo" architecture: a Search API to find the URLs, and a Reader API to clean them.

The Data Flow

graph TD;
    A[User Query: Latest NVIDIA H100 Pricing] --> B(SearchCans Search API);
    B --> C{Get Top 3 URLs};
    C --> D(SearchCans Reader API);
    D -- Convert to Markdown --> E[Clean Context];
    E --> F[LLM / Vector DB];

This approach ensures that your RAG pipeline is fed only high-density information, stripped of ads, popups, and boilerplate.

Top Tools for URL-to-Markdown Conversion

Based on our analysis in the URL to Markdown API Benchmark 2026, here is how the top tools stack up for RAG pipelines.

SearchCans Reader API

Designed specifically for the AI Agent era.

Key Advantage

It is a "Hybrid" API. You can perform a Google Search and Convert to Markdown using the same API key and wallet.

Format

Optimized Markdown that preserves headers (#, ##) and tables, which LLMs love.

Pricing

Pay-As-You-Go (no monthly expiry).

Jina Reader

A popular choice in the open-source community.

Pros

Good support for simple pages; offers a free tier.

Cons

Aggressive rate limits on the free plan; often struggles with heavy JavaScript sites compared to a full browser-based scraper.

Firecrawl

Excellent for crawling entire subdomains.

Pros

Can "crawl" deep into a site to build a knowledge base.

Cons

Expensive subscription model. Overkill if you just need to read one specific page from a search result.

Comparison: Data Density

Implementation: The "Search-to-Markdown" Python Script

The following script automates this workflow. We will use the SearchCans API to find a page and then immediately convert it to Markdown.

Prerequisites

Before running the script, ensure you have:

• Python 3.x installed
• A SearchCans API Key (Free credits available)

Python Implementation: Search-to-Markdown Pipeline

This script demonstrates the complete workflow from search query to clean Markdown output.

# src/pipelines/search_to_markdown.py
import requests
import json

# Configuration
API_KEY = "YOUR_SEARCHCANS_API_KEY"
BASE_URL = "https://www.searchcans.com/api"

def get_markdown_from_search(query):
    """
    1. Searches Google for the query.
    2. Takes the top result.
    3. Converts that result into Clean Markdown.
    """
    headers = {
        "Authorization": f"Bearer {API_KEY}",
        "Content-Type": "application/json"
    }

    # --- Step 1: Search (SERP API) ---
    print(f"🔎 Searching Google for: '{query}'...")
    
    # SERP API payload: s=query, t=engine, d=timeout(ms), p=page
    search_payload = {
        "s": query,
        "t": "google",
        "d": 10000,   # 10s API processing timeout in ms
        "p": 1
    }
    
    try:
        serp_resp = requests.post(
            f"{BASE_URL}/search", 
            headers=headers, 
            json=search_payload,    # Use json= not data=json.dumps()
            timeout=15
        )
        serp_data = serp_resp.json()
        
        if serp_data.get("code") != 0:
            return f"Error: {serp_data.get('msg')}"
            
        # Get the first organic result
        organic_results = serp_data.get("data", [])
        if not organic_results:
            return "No results found."
            
        # Extract URL (supports both 'url' and 'link' keys)
        top_item = organic_results[0]
        top_url = top_item.get('url') or top_item.get('link')
        print(f"🔗 Found Top URL: {top_url}")

    except Exception as e:
        return f"Search Request Failed: {str(e)}"

    # --- Step 2: Read (Reader API) ---
    print(f"📄 Converting to Markdown...")
    
    # Reader API payload: s=url, t=type, w=wait(ms), mode=0/1, d=timeout(ms)
    reader_payload = {
        "s": top_url,
        "t": "url",
        "w": 3000,    # Wait 3000ms for dynamic content (React/Vue)
        "mode": 1,    # 1=headless browser (was b:True, now deprecated)
        "d": 15000    # 15s timeout for JS-heavy pages
    }

    try:
        read_resp = requests.post(
            f"{BASE_URL}/url", 
            headers=headers, 
            json=reader_payload,    # Use json= not data=json.dumps()
            timeout=30
        )
        read_data = read_resp.json()
        
        if read_data.get("code") == 0:
            data_content = read_data.get("data", {})
            
            # Handle potentially stringified JSON in 'data'
            if isinstance(data_content, str):
                try:
                    data_content = json.loads(data_content)
                except:
                    pass
            
            if isinstance(data_content, dict):
                markdown_content = data_content.get("markdown", "")
                return markdown_content
            return str(data_content)
        else:
            return f"Reader Error: {read_data.get('msg')}"

    except Exception as e:
        return f"Reader Request Failed: {str(e)}"

if __name__ == "__main__":
    # Example: Ask a question that requires live data
    topic = "latest spacex starship launch results"
    result = get_markdown_from_search(topic)
    
    print("\n--- LLM Context Data (Markdown) ---\n")
    print(result[:1000])  # Print first 1000 chars
    print("\n... (content continues)")

Pro Tip: Headless Browser Mode

Simple requests libraries fail on React/Next.js websites because the content renders via JavaScript. Set "mode": 1 (headless browser) and "w": 3000 (wait 3000ms) in the Reader payload. This forces a headless browser to render the page before extraction. Note: "b": True is the deprecated parameter — it no longer works and will cause dynamic pages to return empty content.

Parallel Pipeline: Fetch 5 Results in 3 Seconds

The single-URL pipeline above is clean for prototyping but slow for production. Most RAG queries need the top 3–5 results for reliable context. Reading them sequentially adds 3–5s latency per request.

With concurrent.futures.ThreadPoolExecutor, you can read all 5 results simultaneously — reducing total pipeline latency from ~15s to ~3s (Pro plan, June 2026, US region):

import concurrent.futures
import requests

API_KEY = "YOUR_SEARCHCANS_KEY"
BASE_URL = "https://www.searchcans.com/api"

def fetch_markdown(url: str, headers: dict) -> dict:
    """Fetch a single URL as Markdown. Returns {url, markdown} dict."""
    payload = {
        "s": url,
        "t": "url",
        "mode": 0,     # mode:0 = standard HTTP, faster for static content
        "d": 10000
    }
    try:
        resp = requests.post(f"{BASE_URL}/url", headers=headers, json=payload, timeout=15)
        data = resp.json()
        if data.get("code") == 0:
            page = data.get("data", {})
            return {"url": url, "markdown": page.get("markdown", ""), "title": page.get("title", "")}
        return {"url": url, "markdown": "", "error": data.get("msg")}
    except Exception as e:
        return {"url": url, "markdown": "", "error": str(e)}

def parallel_search_to_markdown(query: str, top_n: int = 5) -> list:
    """Full pipeline: search → fetch top N URLs in parallel."""
    headers = {"Authorization": f"Bearer {API_KEY}"}
    
    # Step 1: SERP search
    serp_resp = requests.post(
        f"{BASE_URL}/search",
        headers=headers,
        json={"s": query, "t": "google", "d": 10000, "p": 1},
        timeout=15
    )
    serp_data = serp_resp.json()
    if serp_data.get("code") != 0:
        return []
    
    urls = [r["url"] for r in serp_data.get("data", [])[:top_n] if r.get("url")]
    
    # Step 2: Parallel Reader API calls
    results = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=top_n) as executor:
        futures = {executor.submit(fetch_markdown, url, headers): url for url in urls}
        for future in concurrent.futures.as_completed(futures):
            results.append(future.result())
    
    return [r for r in results if r.get("markdown")]

if __name__ == "__main__":
    import time
    start = time.time()
    pages = parallel_search_to_markdown("NVIDIA H100 GPU price 2026", top_n=5)
    elapsed = time.time() - start
    print(f"Fetched {len(pages)} pages in {elapsed:.1f}s")
    for p in pages:
        words = len(p['markdown'].split())
        print(f"  [{words} words] {p['title'][:60]}")

This parallel approach consumes 10 Reader API credits total (5 URLs × 2 credits each at mode: 0) — approximately $0.0056 per multi-page search at Ultimate plan pricing. The resulting context is rich enough for most factual RAG queries.

Frequently Asked Questions

Q: Why use Markdown instead of Plain Text for RAG context?

A: Markdown preserves structural hierarchy (headers, tables, lists) that plain text flattens. When an LLM reads a table in Markdown, it understands the relationship between rows and columns, making complex financial or technical data intelligible. Plain text loses this structure, reducing retrieval accuracy in structured data scenarios. Our benchmark of 100 pages showed Semantic Markdown preserving 98% of informational content versus 85% for plain text extraction.

Q: What is the correct Reader API parameter for headless browser mode?

A: Use "mode": 1 in the payload. The old "b": True or "b": "true" parameter is deprecated and silently ignored — pages that require JavaScript rendering will return empty or partial content. mode: 0 (2 credits) handles standard HTTP pages; mode: 1 (4 credits) uses a headless Chromium browser for SPAs, dynamic dashboards, and paywalled-adjacent content.

Q: What is the d parameter in the SearchCans API?

A: d is the API-side processing timeout in milliseconds — not a page count or result limit. d: 10000 gives the API a 10-second processing budget. Setting d: 10 creates a 10ms timeout that always fails. For headless browser mode (mode: 1) on slow pages, use d: 15000 or higher.

Q: How does this pipeline compare to using Firecrawl or Jina for RAG?

A: Firecrawl excels at crawling entire subdomains to build comprehensive knowledge bases but runs on a subscription model ($16–$333/month depending on plan). Jina Reader is open-source-friendly but has rate limits that restrict bulk processing. SearchCans Reader API is pay-as-you-go at 2 credits per page ($0.00112 at Ultimate), with no monthly commitment. For a RAG pipeline reading 500 pages/day, that’s approximately $0.56/day — versus $16+/month minimum for Firecrawl.

SearchCans is NOT for crawling an entire website’s sitemap to build an offline knowledge base — that use case is better served by Firecrawl or a custom Scrapy spider. SearchCans Reader API is the right tool for real-time RAG: fetching specific, live URLs returned by a search query at the moment the user asks a question.

Conclusion

Building a RAG pipeline on static data is like driving with a 1990 map. To build AI Research Agents that genuinely provide value, you need to connect them to the live internet.

The Search-to-Markdown pipeline — SERP API for discovery, Reader API for extraction — is the missing architectural link between a user’s question and a grounded, accurate LLM response. With parallel fetching, it runs in under 3 seconds end-to-end.

Get your API Key and start building →