Convert a URL to Markdown in Python and Preserve Source Structure

Converting a URL to Markdown is useful when the returned text preserves the content your application needs. Keep the source URL and access time with the output, then inspect headings, tables and code before chunking it. A smaller string is not automatically a better source.

Make a fresh conversion request

import os
import requests

url = "https://example.org/permitted-article"
response = requests.post(
    "https://www.searchcans.com/api/v1/url",
    headers={"Authorization": "Bearer " + os.environ["SEARCHCANS_API_KEY"]},
    json={"t": "url", "s": url, "mode": 0, "maxCache": 0, "d": 30000},
    timeout=40,
)
response.raise_for_status()
body = response.json()
if body.get("code") != 0 or not isinstance(body.get("data"), dict):
    raise RuntimeError("Reader did not return a successful data object")
markdown = body["data"].get("markdown")
if not isinstance(markdown, str) or not markdown.strip():
    raise ValueError("No usable Markdown returned")
print(markdown)

The URL above is a placeholder; replace it with a source you are permitted to read. This example does not make an arbitrary example URL into a guaranteed live page. Browser-rendered content has a separate mode and completeness check.

Check structure against the source

Do not turn conversion into an accuracy claim

The owned HTML fixture used for this revision retained the required article text, table and code in the candidate Markdown. It is one controlled example, not a whole-web benchmark. If token cost matters, measure both outputs with the tokenizer used by your model and check whether useful evidence was lost.

To measure representation cost, use the paired HTML and Markdown token test and report missing content beside the counts.

Use the URL-to-Markdown evaluation guide when choosing a provider, and the ingestion contract when deciding whether an extracted document is safe to index. This page completes the conversion and inspection task; it does not rank every extraction vendor.

Reference documentation: www.searchcans.com — /reader-api.

Preserve a batch conversion manifest

For a batch of permitted URLs, keep one record per source rather than join every response into a single untraceable string. The source-preserving manifest below retains successful text, a content hash and observation time, while failed reads remain separate. Its structure_review flag stays unreviewed until the earlier structure checks have been applied.

import hashlib
from datetime import datetime, timezone
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urlparse

def batch_manifest(urls, extract, authorized_hosts, current_lanes=1):
    if current_lanes < 1:
        raise ValueError("Confirm available account capacity first")
    ordered = list(dict.fromkeys(urls))
    for url in ordered:
        parsed = urlparse(url)
        if parsed.scheme != "https" or parsed.hostname not in authorized_hosts:
            raise ValueError("Out-of-scope source")
    def convert(url):
        try:
            text = extract(url)
            if not isinstance(text, str) or not text.strip():
                raise ValueError("No usable Markdown")
            return {"url": url, "status": "extracted", "markdown": text,
                "body_sha256": hashlib.sha256(text.encode()).hexdigest(),
                "observed_at": datetime.now(timezone.utc).isoformat(),
                "structure_review": "not_reviewed"}
        except Exception as error:
            return {"url": url, "status": "failed",
                    "error_type": type(error).__name__, "markdown": ""}
    with ThreadPoolExecutor(max_workers=min(3, current_lanes)) as executor:
        return list(executor.map(convert, ordered))

if __name__ == "__main__":
    import json
    rows = batch_manifest(["https://example.test/a", "https://example.test/b"],
        lambda url: "# Synthetic source\n\nRequired detail", {"example.test"}, 2)
    print(json.dumps({"fixture_only": True, "manifest": rows}, indent=2))

Use an extraction callback that validates the HTTP and business response, reads data.markdown and has a finite timeout. Set current_lanes from observed available capacity; the local worker count is capped at three. This bounds in-flight calls, not the number of submitted URLs or total credits. The caller must separately cap the input list and budget before collecting it.

Persist the manifest after collection and retry only reviewed failed inputs. The function is not a durable per-item checkpoint writer and does not prove that text or RAG answers are correct. Use the Reader-to-RAG contract for downstream loading rather than expand this conversion page into another generic pipeline tutorial.

Related implementation and validation guides

• How to Choose a URL-to-Markdown API for RAG