A research API stack needs more than a list of vendors. It needs a contract for discovering sources, reading them, recording what was observed and deciding which claims they support. Keep failed reads and unreviewed sources visible. A returned snippet or a successful extraction is not a verified research conclusion.
Choose components from the research deliverable
| Required output | Component | What it must preserve |
|---|---|---|
| Candidate sources | Search API | Query, market, URLs and sampling time |
| Page evidence | URL or document extractor | Source URL, body and retrieval time |
| Reproducible record | Your evidence ledger | Body hash, status and claim-review state |
| Updated conclusion | Review and recheck process | What changed and which claims need another check |
Add specialist APIs only when the task needs their data. A vector database is useful for indexing stored evidence; it is not proof that the evidence is current or supports a conclusion. An LLM can propose a summary, but the source-to-claim review must remain a separate step.
A bounded search-to-source ledger
The example uses SearchCans search to find candidates and Reader to extract at most two approved-host URLs. It records read failures separately. Install requests and set SEARCHCANS_API_KEY. The allowlist is an application policy, and the example makes no claim that every permitted URL can be extracted.
import os
import requests
from datetime import datetime, timezone
from urllib.parse import urlsplit, urldefrag
def organic(envelope):
if envelope.get("code") != 0:
raise ValueError("Search business error")
data = envelope.get("data")
if not isinstance(data, dict) or not isinstance(data.get("organic"), list):
raise ValueError("Expected data.organic array")
return [row for row in data["organic"] if isinstance(row, dict)
and isinstance(row.get("link"), str)
and urlsplit(row["link"]).scheme == "https"
and urlsplit(row["link"]).hostname
and not urlsplit(row["link"]).username]
def call(path, payload, session=requests, timeout=40):
response = session.post("https://www.searchcans.com/api/v1/" + path,
headers={"Authorization": "Bearer " + os.environ["SEARCHCANS_API_KEY"]},
json=payload, timeout=timeout)
response.raise_for_status()
envelope = response.json()
if envelope.get("code") != 0 or not isinstance(envelope.get("data"), dict):
raise ValueError("Business or response-schema error")
return envelope
import hashlib
def collect_research_sources(query, allowed_hosts, session=requests):
hits = organic(call("search", {"t":"google", "s":query,
"country":"us", "language":"en", "d":30000}, session))
records = []
urls = list(dict.fromkeys(row["link"] for row in hits
if urlsplit(row["link"]).hostname in allowed_hosts))[:2]
for url in urls:
record = {"source_url":url, "attempted_at":datetime.now(timezone.utc).isoformat()}
try:
envelope = call("url", {"t":"url", "s":url, "mode":0,
"proxy":0, "maxCache":0, "d":30000}, session)
markdown = envelope["data"].get("markdown")
if not isinstance(markdown, str) or not markdown.strip():
raise ValueError("Empty source text")
record.update(status="ok", text=markdown, retrieved_at=datetime.now(timezone.utc).isoformat(),
body_sha256=hashlib.sha256(markdown.encode()).hexdigest(),
claim_review="not_reviewed")
except (requests.RequestException, ValueError):
record.update(status="read_failed", claim_review="not_eligible")
records.append(record)
return records
Record support, not just source count
For each assertion, add a claim ID, the source record it relies on, the relevant passage and one of supports, contradicts or not established. The example starts with not_reviewed on purpose. A SHA-256 value lets you detect that the extracted body changed; it does not authenticate the publisher or certify the statement.
| Failure | Research consequence |
|---|---|
| Search succeeds but no allowed URL appears | Report no eligible candidates |
| Reader returns an empty body | Exclude that source from claim support |
| One of two reads fails | Keep the successful source and the missing coverage separately |
| Body hash changes on recheck | Review affected claims before reusing the conclusion |
A small verification task you can repeat
Use two synthetic documentation records with one read failure. Assert that only the successful body receives a hash and that neither record is marked as reviewed support. Then change one body string and check that its hash changes. These are provenance and update-detection tests, not a benchmark of factual accuracy.
Decide retention and access rules for the evidence your system stores. Do not assume that a managed API removes your responsibility for those choices. For evaluation use the source-evidence quality method. For traceable repeating jobs use the monitoring ledger. the agent grounding architecture covers the larger tool loop; this page owns the research deliverable and its evidence record.