A cited answer can still be wrong. Evaluate grounding by connecting each material claim to a permitted source, checking the passage and its context, and resolving conflicting or stale evidence. A valid URL or an exact quote is necessary in some workflows, but neither proves that the claim follows from the evidence.
Define what a source is allowed to support
Record source ID, URL, retrieved version, reviewed passage, permission decision and the date through which the review remains usable for this task. A software API field may need the current official documentation; a historical claim may need a dated record. Prefer the source that directly supports the claim rather than the page with the strongest brand recognition.
Keep mechanical checks separate from judgment
from datetime import datetime
def review_grounding(claim, sources, allowed_ids, now):
sid = claim.get("source_id")
source = sources.get(sid)
if sid not in allowed_ids or not isinstance(source, dict):
return {"status":"reject", "reason":"unapproved_source"}
quote, text = claim.get("quote"), source.get("text")
if not isinstance(quote, str) or not quote.strip() or not isinstance(text, str) or quote not in text:
return {"status":"reject", "reason":"quote_not_found"}
expiry = datetime.fromisoformat(source["review_until"])
if expiry.tzinfo is None or now.tzinfo is None:
raise ValueError("Timezone-aware review times required")
if now > expiry:
return {"status":"review", "reason":"stale_review"}
if claim.get("conflicting_source_ids"):
return {"status":"review", "reason":"source_conflict"}
return {"status":"needs_entailment_review", "source_id":sid}
if __name__ == "__main__":
source = {"text":"Protocol version 2 is current.", "review_until":"2026-10-12T00:00:00+00:00"}
print(review_grounding({"source_id":"s1","quote":"version 2"}, {"s1":source}, {"s1"},
datetime.fromisoformat("2026-10-11T00:00:00+00:00")))
The synthetic fixture returns needs_entailment_review, not “true.” The quote exists and the source passes the configured checks, but a reviewer must still decide whether it supports the proposed statement. An unapproved source is rejected. An expired review or a recorded conflicting source is held for review. The expiry is a task policy, not proof of a source’s publication freshness.
Build evaluation cases that expose the boundary
| Case | Expected result |
|---|---|
| Real quote used for a stronger claim | Human entailment review should reject the overstatement |
| Quote absent from the saved version | Mechanical rejection |
| Two approved sources disagree | Conflict review, with both versions preserved |
| Correct answer from an unapproved source | Review permission before use |
| Current citation beside an unrelated claim | Reject unsupported association |
Label cases before running the system and keep reviewer disagreements. Measure unsupported assertions and missing evidence separately from citation formatting. Additional URLs alone do not demonstrate better answer accuracy. Search discovers candidates; page reading obtains passages; the evidence review connects them to an answer.
For Claude’s native search integration, use Claude web-search citation acceptance; a citation remains evidence to review, not proof of the answer.
Use the agent architecture for access policy and the RAG pipeline for ingestion. This page owns claim-to-evidence evaluation rather than a general SEO or search-provider ranking.
