Quick answer
Choose a URL-to-Markdown API for RAG by testing it against the pages you actually need to ingest. Check the main text, heading structure, source metadata, JavaScript rendering, document handling, failures, and cost. A clean demo is not enough. Your retrieval pipeline needs predictable output when real source pages change.
What a URL-to-Markdown API must do for RAG
A RAG pipeline needs usable source material before it needs embeddings or a vector store. HTML can contain navigation, cookie banners, repeated cards, scripts, and layout markup that do not belong in a retrieval corpus. A URL-to-Markdown API should retrieve a permitted public URL and return text that your ingestion process can inspect, chunk, attribute, and reprocess.
The useful question is not “Can this API turn a page into Markdown?” Most tools can do that on a simple static page. Ask whether the result preserves the information a user would need to answer a question and whether your pipeline can tell when the result is incomplete.
Start with an ingestion contract
Define the output your pipeline needs before comparing tools. For each source URL, decide what you will retain and what makes an extraction acceptable.
Main content
Retrieval quality suffers when navigation or repeated templates dominate chunks. Compare the extracted opening, headings, and conclusion with the source page.
Structure
Headings, lists, tables, and code help you chunk and cite content consistently. Verify that a representative article keeps meaningful headings and code blocks.
Source identity
A retrieved answer needs a traceable source. Store the requested URL, extraction time, API result code, and a content fingerprint in your own pipeline.
Dynamic pages
Some sites only render useful content after client-side code runs. Test both the normal request and browser rendering on a page you are allowed to process.
Documents
PDF and office-document ingestion has different output and failure cases. Test a real PDF or DOCX that reflects your expected corpus.
Failure behavior
A blank or partial response should not silently enter the index. Deliberately test a slow page, an unavailable page, and a page with little visible text.
Cost
Reprocessing, retries, and browser rendering change the total cost. Estimate from your own URL mix, not from a single request.
This contract also makes vendor changes easier to detect. Keep a small regression set of source URLs and rerun it after changing an extraction setting or SDK version.
Build a representative test set
Do not evaluate an extraction API on one popular news article. A useful test set contains pages that resemble the corpus you plan to search:
- A static technical article with headings, lists, and code.
- A documentation page that has a navigation shell and a substantial main section.
- A JavaScript-rendered page that you have permission to fetch.
- A table-heavy page whose rows matter to retrieval.
- A PDF or office document if your application accepts documents.
- A page that fails, redirects, or returns almost no useful visible text.
For every test, save the source URL and manually review a few chunks. Check whether each chunk would still make sense when an LLM sees it without the original page around it. A Markdown response can be valid syntax and still be poor retrieval input.
Test standard extraction before browser rendering
Browser rendering is useful when an allowed target needs client-side rendering, but it should not be the automatic first setting for every URL. Start with the standard request, then use browser rendering only when the returned main content is incomplete for the target page.
With SearchCans Reader API, a normal URL extraction uses POST /api/v1/url with t: "url" and the target URL in s. Set mode: 1 only when you need headless browser rendering. The optional w value is a post-load wait in milliseconds and applies when browser mode is used. Test it against a real page instead of choosing a long wait by habit.
import os
import requests
api_url = "https://www.searchcans.com/api/v1/url"
target_url = "https://example.com/article"
response = requests.post(
api_url,
headers={
"Authorization": f"Bearer {os.environ['SEARCHCANS_API_KEY']}",
"Content-Type": "application/json",
},
json={
"t": "url",
"s": target_url,
},
timeout=30,
)
response.raise_for_status()
result = response.json()
if result.get("code") != 0:
raise RuntimeError(result.get("msg", "Reader request failed"))
markdown = result["data"].get("markdown", "")
if not markdown.strip():
raise RuntimeError("Reader returned no Markdown")
print(markdown[:500])If the page needs browser rendering, add "mode": 1 to the request and compare the two outputs. Do not treat browser rendering or proxy escalation as permission to fetch content that your application is not allowed to access.
Check documents separately from web pages
URLs that point to PDF, DOCX, XLSX, or PPTX files should have their own test cases. In SearchCans, file parsing uses the same Reader endpoint with file: 1; parsed document Markdown is returned in data.fileMarkdown. That is a different contract from a web-page response in data.markdown.
Keep document processing separate in your ingestion code. It lets you record document-specific failures, preserve the original file URL, and avoid silently treating a missing fileMarkdown value as a successful page extraction. See the File Extraction API for the supported document workflow.
Make incomplete extraction visible
An extraction API can return HTTP success while the result is still wrong for your use case. Your pipeline should reject or route for review when it sees conditions such as:
- empty Markdown after whitespace is removed
- a response dominated by repeated navigation or cookie text
- missing expected headings on a known structured source
- a document response without the expected document content
- an API error code, timeout, or a target URL outside your allowed source policy
Log those outcomes with the source URL and the extraction timestamp. When a source changes its template, the logs give you a way to reprocess affected documents instead of guessing which chunks became stale.
Estimate cost from the workflow, not a headline rate
Cost depends on the request mix. For SearchCans, a standard Reader request costs 2 credits. Browser mode, proxy tiers, document parsing, and retries should be measured against your own source set before you commit to an ingestion design. The pricing page is the current source for plan and credit details.
Calculate at least these categories:
- first-pass extraction for every new source
- scheduled refreshes for sources that change
- retry volume from failures that you decide are worth retrying
- browser-rendered requests for pages that genuinely need them
- document extraction if your corpus includes files
This estimate is more useful than a comparison table full of fixed competitor claims. It shows where your own system spends requests and where a cache, a freshness rule, or a source-specific policy will help.
A practical rollout for a RAG ingestion pipeline
- Define the source policy and retain only URLs you are permitted to process.
- Build the small regression set described above.
- Run standard extraction and inspect the returned Markdown with a human review sample.
- Add browser rendering only for targets where the standard output misses needed content.
- Treat documents as a separate path and check
fileMarkdownbefore indexing. - Store source provenance and a content fingerprint with each indexed chunk.
- Re-run the regression set after a parser, source template, or extraction-setting change.
SearchCans combines URL extraction, file extraction, and web search under one account. For implementation details, use the Reader API, test an allowed URL in the Playground, and review the current pricing before sizing a production workload.
FAQ
Q: How should I choose a URL-to-Markdown API for RAG?
A: Define an ingestion contract, then test each API on static pages, dynamic pages, structured documentation, and documents that match your real corpus. Review main content, structure, error handling, and source provenance before comparing workflow cost.
Q: Should every RAG URL use browser rendering?
A: No. Start with standard extraction and use browser rendering when a permitted page needs client-side rendering for the content you need. Compare the output, because rendering adds work without guaranteeing a better extraction for every page.
Q: What should I retain with extracted Markdown?
A: Retain the requested source URL, extraction time, API result status, and a content fingerprint in your own pipeline. Those records help you trace retrieved answers and reprocess content after a source changes.
Q: Can one URL-to-Markdown workflow handle PDFs as well as web pages?
A: It can, but document extraction should be tested and logged separately. With SearchCans, use file: 1 for document parsing and read data.fileMarkdown rather than assuming a web-page data.markdown response.
