AI agents need a web-search tool when a task depends on current or source-verifiable information. A dependable implementation separates planning, search, page selection, extraction, evidence checking, and answer generation. It also stops when the evidence is sufficient instead of letting the model search indefinitely.
The Workflow at a Glance
- Classify whether the request needs current web evidence.
- Turn the user request into one or more focused search queries.
- Call a SERP API and retain the returned title, URL, snippet, and position.
- Select a small set of promising URLs before extracting full pages.
- Convert selected pages to Markdown with a Reader API.
- verify dates, source identity, and agreement between sources.
- Generate an answer whose material claims point to the retained URLs.
- Stop when the answer is supported or report that the evidence is insufficient.
This page owns the runnable search-to-evidence agent workflow. Concurrency planning is covered separately in Scaling AI Agents with Parallel Lanes.
Define the Agent’s Tool Contract
The model should not receive a vague browse_the_web function. Give it narrow tools with explicit inputs and outputs:
| Tool | Input | Output the agent should retain |
|---|---|---|
search_web |
query, engine, country, language, page | title, URL, snippet, result position |
read_url |
URL, render mode, timeout, proxy choice | Markdown, final URL, retrieval time, status |
finish_research |
supported claims and source URLs | final evidence set and unresolved questions |
The search tool discovers sources. The Reader tool extracts evidence from selected pages. Keeping the two steps separate makes cost, failures, and citations easier to audit.
Run a Minimal Search-to-Reader Pipeline
The example below uses the current SearchCans v1 endpoints. It limits extraction to the first three selected results and preserves the source URL beside the Markdown.
import os
import requests
API_KEY = os.environ["SEARCHCANS_API_KEY"]
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
SEARCH_URL = "https://www.searchcans.com/api/v1/search"
READER_URL = "https://www.searchcans.com/api/v1/url"
def call_api(endpoint, payload, timeout):
response = requests.post(endpoint, json=payload, headers=HEADERS, timeout=timeout)
response.raise_for_status()
result = response.json()
if result.get("code") != 0:
raise RuntimeError(result.get("message", "SearchCans request failed"))
return result["data"]
def research(query, max_pages=3):
search_data = call_api(
SEARCH_URL,
{"s": query, "t": "google", "p": 1},
timeout=15,
)
organic = search_data.get("organic", search_data if isinstance(search_data, list) else [])
evidence = []
for result in organic[:max_pages]:
source_url = result.get("link") or result.get("url")
if not source_url:
continue
page = call_api(
READER_URL,
{"s": source_url, "t": "url", "mode": 1, "w": 3000, "d": 30000, "proxy": 0},
timeout=35,
)
markdown = page.get("markdown", "")
if markdown.strip():
evidence.append({
"title": result.get("title", ""),
"source_url": source_url,
"markdown": markdown,
})
return evidenceInspect one live response in the Playground before binding production code to a result field. Store the raw response during development so a schema change or empty result can be diagnosed without guessing.
Add Evidence and Citation Rules
A web-enabled agent can still produce unsupported claims. Apply these rules after extraction:
- prefer original documentation, official announcements, primary research, and first-party datasets;
- record the publication or update date when freshness matters;
- keep the source URL attached to each extracted passage;
- require two independent sources for a disputed claim when feasible;
- label inference separately from what the source directly states;
- omit a numerical or comparative claim when the evidence cannot be traced;
- never cite a search snippet as though the full page had been checked.
The agent should build a small evidence ledger before drafting: claim, source_url, source_date, retrieved_at, supporting_passage, and support_type. This makes the final answer reviewable and gives downstream systems a stable citation object.
Prevent Infinite Search Loops
Use explicit limits rather than asking the model to decide forever:
| Limit | Example behavior |
|---|---|
| Query budget | Stop after a fixed number of search calls |
| Extraction budget | Read only the URLs most likely to answer the question |
| Duplicate rule | Skip a URL or domain already processed unless a second page has a distinct job |
| Evidence threshold | Stop once every material claim has adequate support |
| Time budget | Return partial findings and unresolved questions when the deadline is reached |
| Failure budget | Stop escalating proxy modes after the allowed retry path is exhausted |
SearchCans charges 1 credit for a successful standard Google or Bing SERP request. A standard Reader request costs 2 credits; proxy modes add 2, 5, or 10 credits. Failed or non-200 requests are not billed. Use these current product facts to calculate a request budget, but read plan prices and Parallel Lanes from the live pricing page.
Handle Failures Without Hiding Them
Classify failures so the agent can choose a bounded next step:
| Failure | Next action |
|---|---|
| Empty or irrelevant SERP | Rewrite the query once, then stop or ask for a narrower task |
| Timeout | Retry with backoff inside the remaining time budget |
| JavaScript page lacks content | Retry Reader with mode: 1 |
| Access challenge | Escalate to an approved proxy mode and record the added cost |
| Empty Markdown | Mark extraction failed; do not give the LLM an empty evidence object |
| Conflicting sources | Prefer primary evidence and surface the disagreement |
Do not turn every failure into another model call. The orchestrator should enforce budgets and return a structured reason when research cannot finish.
Decide When Web Search Is the Wrong Tool
Skip live search when the answer is already contained in trusted application data, when the task is purely transformative, or when the user supplied the complete source. Use a database or product API for precise transactional facts. Use browser automation when the task requires interaction rather than retrieval. A SERP plus Reader workflow is best for discovering public sources and extracting their readable content.
Frequently Asked Questions
Q: Does web access eliminate hallucinations?
A: No. Web access supplies current evidence, but the model can still misread a source or make an unsupported inference. Keep claim-level citations and validate important conclusions.
Q: Should the agent read every search result?
A: No. Rank candidates from titles, snippets, domains, dates, and task fit, then extract a small evidence set. Reading every result increases cost and noise.
Q: How should sources be cited?
A: Retain the original URL and a supporting passage during research. Generate the final answer from that evidence ledger and omit claims that have no traceable support.
Q: Which framework is required?
A: None. LangChain, LlamaIndex, CrewAI, and model-specific tool APIs can all orchestrate the same search, read, verify, and stop sequence. The tool contract and evidence rules matter more than the framework name.
