LLM web crawler API guide for content extraction and RAG

Quick answer

An LLM web crawler is a workflow, not a single API. It discovers permitted sources, extracts useful content, records where each chunk came from, validates the result, and refreshes changed pages. For RAG, the goal is reliable and traceable source material, not simply collecting the most HTML.

What makes a web crawler useful to an LLM

A browser can display a page that is a poor input for retrieval. Navigation, cookie banners, repeated cards, scripts, and product chrome can overwhelm the text that a user would actually want to retrieve. An LLM-ready crawler needs a clear boundary between finding sources, extracting a selected URL, and preparing those results for search or generation.

Markdown helps because headings, lists, tables, and code can survive the extraction step in a form that is easier to inspect and chunk. It does not guarantee that a model will give a correct answer. The application still needs source selection, retrieval evaluation, and citations or provenance where they matter.

Start with a source policy

Decide what the crawler is allowed to process before choosing an extraction tool. A source policy should describe the domains, URL patterns, document types, refresh schedule, and access rules that apply to your workload. It should also define what you will do with redirects, unavailable pages, and pages that return little useful content.

This is where a production crawler differs from a broad scraping experiment. A URL-to-Markdown API can extract a URL you submit, but it does not replace your own queue, crawl frontier, domain allowlist, or compliance review. Only send URLs your application is permitted to access and store.

Separate discovery from extraction

Search discovery and page extraction answer different questions:

Search answers: which current pages are relevant to this request?
Extraction answers: what useful text and structure does this selected URL contain?
Validation answers: is this result good enough to index or show to a user?

With SearchCans, POST /api/v1/search returns Google or Bing search results. POST /api/v1/url is the Reader endpoint for extracting a chosen URL. Keeping those stages separate gives you a practical audit trail: you can explain why a page entered the corpus and reproduce the extraction when the source changes.

Build a small ingestion contract

Define the fields and quality checks your system expects from every accepted result. The contract can be simple, but it should be explicit.

Source identity

Store the URL your system requested, the extraction timestamp, the API result status, and a content fingerprint you calculate after extraction. This lets you trace a retrieved answer to the source and detect when the stored content changes.

Main content and structure

Review representative output by hand. A useful result preserves meaningful headings, lists, tables, and code where those elements matter. Reject output that is blank, dominated by template text, or missing the section your pipeline expected to find.

Refresh behavior

Use the source type to decide how often to check it again. A release note, pricing page, or policy may need a different refresh schedule from a historical article. Record the last successful extraction so your system can avoid treating old text as current evidence.

Failure behavior

Do not silently index an empty Markdown response. Log HTTP failures, API error codes, timeouts, redirects, and content-quality failures. Those records make it possible to retry the right sources and keep poor extractions out of retrieval.

Use browser rendering only when the source needs it

Start with standard URL extraction. If the useful content is missing because a permitted target relies on client-side rendering, run a test with browser rendering and compare the result. In SearchCans Reader API, mode: 1 enables headless browser rendering. The optional w value controls the post-load wait in milliseconds and applies when browser mode is used.

Browser rendering is a tool for page rendering, not an access override. It does not make a disallowed source permissible, and a longer wait does not prove that the extracted content is complete. Keep a regression set of dynamic pages and compare the output after changing a wait value or render setting.

Handle documents as a separate path

PDFs and office documents have different output and failure conditions from web pages. SearchCans uses the same Reader endpoint for document parsing with file: 1; document Markdown is returned in data.fileMarkdown. A normal web-page extraction returns Markdown in data.markdown.

Keep those responses separate in your ingestion code. A missing fileMarkdown value is not a successful web-page result, and a document should retain its original file URL and extraction record. The File Extraction API documents the file-specific workflow.

A minimal SearchCans search-to-extraction flow

This example keeps discovery and extraction separate. It validates that search returned a URL, then refuses to continue when Reader returns an error or empty Markdown.

import os

import requests

headers = {
    "Authorization": f"Bearer {os.environ['SEARCHCANS_API_KEY']}",
    "Content-Type": "application/json",
}

search_response = requests.post(
    "https://www.searchcans.com/api/v1/search",
    headers=headers,
    json={"t": "google", "s": "RAG source evaluation"},
    timeout=30,
)
search_response.raise_for_status()

search_data = search_response.json()
if search_data.get("code") != 0 or not search_data.get("data"):
    raise RuntimeError("Search returned no usable results")

source_url = search_data["data"][0]["url"]

reader_response = requests.post(
    "https://www.searchcans.com/api/v1/url",
    headers=headers,
    json={"t": "url", "s": source_url},
    timeout=30,
)
reader_response.raise_for_status()

reader_data = reader_response.json()
if reader_data.get("code") != 0:
    raise RuntimeError(reader_data.get("msg", "Reader request failed"))

markdown = reader_data["data"].get("markdown", "")
if not markdown.strip():
    raise RuntimeError("Reader returned no Markdown")

print(source_url)
print(markdown[:500])

For a page that needs client-side rendering, add "mode": 1 and test the output against the standard response. Do not add browser rendering to every request by default. Use the Reader API for the current parameter reference and the Playground to test an allowed URL.

Validate the content before indexing

Before creating embeddings or chunks, check whether the extracted result meets the contract. A useful validation step can flag:

blank Markdown after whitespace is removed
repeated site navigation or consent text dominating the response
a missing title or heading that should exist on a known structured source
unexpected redirects or source URLs outside your allowlist
a stale fingerprint when the source should have been refreshed
document responses that do not contain the expected fileMarkdown content

The exact thresholds are workload-specific. The important part is to make a failed extraction observable. That allows an engineer to review it instead of letting a weak result become invisible training or retrieval context.

Plan refreshes and cost from your real URL mix

The recurring cost of an LLM web crawler comes from first-pass extraction, refreshes, retries you decide to make, browser-rendered pages, and document parsing. SearchCans standard Google or Bing search requests use 1 credit, while a standard Reader request uses 2 credits. Check the current pricing details before estimating a production workload.

Measure the request mix from a representative source set. A knowledge base with mostly static documentation behaves differently from a workflow that follows breaking news or frequently changing product pages. This makes a useful capacity plan and avoids treating a single test request as proof of operating cost.

A practical rollout sequence

Write a source policy and an ingestion contract.
Build a small regression set with static pages, dynamic pages, tables, documents, and controlled failures.
Use search only to discover candidate sources, then retain the selected URL with its discovery context.
Extract standard pages first and use browser rendering only where your tests show it is needed.
Validate Markdown before chunking, embedding, or exposing it to an agent.
Store provenance, timestamps, and a fingerprint with every accepted result.
Re-run the regression set after a source template or extraction setting changes.

SearchCans provides the search and extraction building blocks for this workflow, while your application owns source policy, queues, refresh timing, and evaluation. For a more focused URL-to-Markdown selection checklist, see How to choose a URL-to-Markdown API for RAG.

FAQ

Q: Does Markdown extraction prevent LLM hallucinations?

A: No. Cleaner source text can make retrieval easier to inspect, but it does not guarantee a correct model answer. Keep source provenance, evaluate retrieval quality, and cite or verify important claims in the application.

Q: Is a URL extraction API a complete web crawler?

A: No. A URL extraction API processes URLs you submit. A complete crawler system also needs source policy, queues, scheduling, refresh rules, and a way to handle errors and duplicate pages.

Q: Should an LLM web crawler render every page in a browser?

A: No. Use standard extraction first. Enable browser rendering for permitted pages only when testing shows that client-side rendering is required for the content your application needs.

Q: What data should an LLM crawler store with each chunk?

A: Store the source URL, extraction timestamp, API outcome, and a content fingerprint in addition to the chunk. Those fields support citations, debugging, and targeted refreshes when a source changes.