Select RAG Evidence Across Sources Without Counting Copies Twice

A multi-source RAG pipeline needs independent evidence, not just more URLs. Several sites can repeat one original statement, and a later page can quote an older revision. Deduplicate source material and keep dates and conflicts before combining passages for an answer.

Track source identity and content identity

Do not erase document identity during normalization

from urllib.parse import urlsplit, urlunsplit

def normalized_url(url):
    p = urlsplit(url)
    if p.scheme not in ('https', 'http') or not p.hostname:
        raise ValueError('Absolute HTTP URL required')
    # Preserve query/path: they may identify different documents.
    return urlunsplit((p.scheme.lower(), p.netloc.lower(), p.path or '/', p.query, ''))

assert normalized_url("https://EXAMPLE.org/a?lang=en#part") == "https://example.org/a?lang=en"

The local helper removes the fragment but preserves the path and query because they may identify different documents. It rejects non-HTTP input. Production URL normalization needs a policy appropriate to the actual source; removing every query parameter can merge different revisions or languages.

Resolve evidence before generation

Read the relevant source section, check its date and prefer a traceable original for a factual claim. If independent sources disagree, preserve the disagreement and the interpretation rule rather than hiding it in a longer prompt. Use an unresolved answer when the evidence cannot complete the task.

The normalization and citation fixtures test mechanical boundaries. They do not establish that a multi-source answer is more accurate merely because it has more sources.

Reference documentation: docs.python.org — /3/library/urllib.parse.html; www.searchcans.com — /reader-api.

Related implementation and validation guides

• Real-Time RAG Web Search: A Practical Evidence Architecture