Quick answer
A real-time RAG web search API flow uses search only when a question needs current evidence. It selects a small set of sources, extracts the useful content, records URLs and retrieval time, applies freshness rules, and gives the model only context that passed review. Live search is not a substitute for source selection or citations.
When real-time web search belongs in RAG
Not every RAG question needs a new web search. Stable internal documentation, completed projects, and historical material can usually come from a maintained knowledge base. Live search is useful when the answer may depend on a recent release, policy, availability change, price update, news event, or public information that changes more quickly than your local index.
The first decision is therefore a freshness decision. Before calling a search API, ask whether an answer from your existing corpus could be old enough to change the result. That simple check avoids adding latency and unreviewed web content to questions that do not need it.
The real-time RAG loop
A reliable live-data flow has five steps:
- Classify the question and decide whether it needs fresh sources.
- Search for candidate URLs that match the question.
- Select a small number of allowed sources and extract their main content.
- Validate the result, attach provenance, and build the context passed to the model.
- Cache or refresh the result according to the source’s expected rate of change.
The model should not receive a full search result page or every extracted URL. Large, unfiltered context can hide the useful evidence and makes it harder to explain why the answer used a particular source.
Search for candidate sources
SearchCans SERP API uses POST /api/v1/search for Google or Bing results. A search result is a discovery signal, not final evidence. Your application should record the query, filter the returned URLs with its source policy, and select the pages that are appropriate for the user question.
Selection rules can include domain allowlists, source type, date sensitivity, result relevance, duplication, and regional policy. A support application might prefer official documentation. A market-research workflow may accept multiple public sources but still needs to retain the URLs it used.
Extract selected URLs into usable context
Once the application selects a URL, SearchCans Reader API uses POST /api/v1/url with t: "url" and the target URL in s. A normal web-page extraction returns Markdown in data.markdown. Use browser rendering with mode: 1 only when a permitted page requires client-side rendering for the content you need.
Do not treat a successful HTTP response as proof that the extraction is usable. Check for empty Markdown, template-heavy output, missing sections, unexpected redirects, or sources outside the policy. For a detailed URL-extraction evaluation checklist, see How to choose a URL-to-Markdown API for RAG.
A minimal search-to-context example
This example keeps search, selection, and extraction as separate steps. It returns a small set of source records that a later retrieval or generation stage can inspect.
import os
import requests
headers = {
"Authorization": f"Bearer {os.environ['SEARCHCANS_API_KEY']}",
"Content-Type": "application/json",
}
def search_sources(query):
response = requests.post(
"https://www.searchcans.com/api/v1/search",
headers=headers,
json={"t": "google", "s": query},
timeout=30,
)
response.raise_for_status()
payload = response.json()
if payload.get("code") != 0:
raise RuntimeError(payload.get("msg", "Search failed"))
return payload.get("data") or []
def extract_source(url):
response = requests.post(
"https://www.searchcans.com/api/v1/url",
headers=headers,
json={"t": "url", "s": url},
timeout=30,
)
response.raise_for_status()
payload = response.json()
if payload.get("code") != 0:
raise RuntimeError(payload.get("msg", "Reader request failed"))
markdown = payload["data"].get("markdown", "")
if not markdown.strip():
raise RuntimeError("Reader returned no Markdown")
return markdown
query = "latest policy changes for a regulated product"
results = search_sources(query)
selected = [item for item in results if item.get("url")][:3]
context = []
for item in selected:
context.append({
"query": query,
"source_url": item["url"],
"markdown": extract_source(item["url"]),
})
print([record["source_url"] for record in context])The list comprehension is intentionally not a ranking policy. Replace it with criteria appropriate to your application before production use. Keep the query, selected URL, retrieval timestamp, and any quality checks with the extracted content.
Preserve citations and source provenance
Every live source should carry enough context to be traced later. At a minimum, retain:
- the user request or search query
- the source URL selected by the application
- the time of extraction
- the API result status and retry decision
- a content fingerprint generated by your application
- the citation text or source reference shown with an answer
These records support auditability and targeted refreshes. They also make it possible to answer a user who asks where a claim came from. Markdown helps retain content structure, but provenance is what connects a response to a real source.
Use freshness rules instead of constant re-searching
Live web retrieval should be selective. Set a refresh rule that reflects how quickly a source can change. A release-note page might need a short refresh window. A technical reference that changes rarely can be reused longer after your application records a successful extraction.
When the freshness window expires, search or re-extract as the workload requires. Do not invent a universal time-to-live. The right interval depends on the source type, the cost of an outdated answer, and whether a user needs an answer about the present moment.
Handle dynamic pages and documents separately
Some web pages require browser rendering; others do not. Test standard extraction first, then add mode: 1 only when the source needs it. The optional w value controls the post-load wait in milliseconds when browser mode is used.
Documents belong in a separate ingestion path. SearchCans File Extraction uses the same endpoint with file: 1 and returns document Markdown in data.fileMarkdown. Validate document output before adding it to the same corpus as web pages. See the File Extraction API for the current document workflow.
Add review gates before generation
Real-time retrieval can supply poor or irrelevant context. Add deterministic checks before sending it to the model:
Source policy
Allow only domains and URL patterns appropriate to the task. Do not treat a search ranking as permission to retrieve or cite a source.
Content checks
Reject empty or template-heavy Markdown, duplicate sources, and pages that fail a task-specific relevance check. A page can be current and still not answer the question.
Answer checks
Require the answer to cite its selected sources when the user needs traceability. For high-impact decisions, route the proposed answer and sources to a human reviewer instead of relying on the model to judge its own certainty.
Estimate the request mix before scaling
SearchCans standard Google or Bing search requests use 1 credit. A standard Reader request uses 2 credits. Your total usage depends on the proportion of questions that need freshness, the number of selected sources per question, retries, browser-rendered pages, and document extraction. Use the current pricing page when sizing a production workflow.
Measure source-selection success, empty extraction rate, freshness lag, retry rate, time to a grounded answer, and the percentage of answers that need review. Those metrics reveal whether the bottleneck is search, extraction, policy, or generation.
Keep the architecture boundary clear
This article covers live retrieval for RAG. It does not give an agent authority to perform state-changing web actions. For a broader design that separates known-URL reading, web research, and authorized actions, see AI agent web access architecture.
FAQ
Q: When should a RAG pipeline use live web search?
A: Use it when the answer depends on information that may have changed since your local corpus was built, such as current releases, policies, availability, news, or time-sensitive public facts. Stable material can remain in a maintained local index.
Q: How many web sources should a real-time RAG request use?
A: Select a small number of sources that match your policy and the user question. More sources are not automatically better. Review relevance, duplication, and whether each source adds evidence before building the model context.
Q: Does a search result snippet count as a citation?
A: No. Use the snippet to discover a candidate URL, then retrieve and retain the selected source before citing it in an important answer.
Q: How can a real-time RAG system keep answers fresh?
A: Record the extraction time and apply refresh rules based on source type and risk. Re-search or re-extract when the stored source is too old for the question, rather than refreshing every document on the same schedule.
