Evaluate Fresh-Web RAG With Frozen Sources and Supported Claims

Evaluate fresh-web RAG against a frozen evidence set before comparing answers. A live source can change between runs, so a higher score may reflect different evidence rather than a better retrieval or generation method. Retain source revisions, question labels and supported claims together.

Freeze the parts needed to reproduce a run

Keep support checks distinct from evaluator opinions

An automatic judge can help prioritize review, but its score is not factual truth. Start with source-linked claim checks and retain the exact passage being used. A quote existing in a document is a useful mechanical check; whether it supports the claim requires context and, for ambiguous cases, human review.

Do not silently refresh the source corpus halfway through a model comparison. If freshness is the variable being tested, run a separate dated experiment and state which source versions each run used.

A minimal local citation gate

def supported_answer(claims, source_map):
    return all(c.get('source_id') in source_map and bool(c.get('quote'))
               and c['quote'] in source_map[c['source_id']] for c in claims) and bool(claims)

sources = {"d1": "The fixture supports version 2."}
assert supported_answer([{"source_id": "d1", "quote": "version 2"}], sources)
assert not supported_answer([{"source_id": "d1", "quote": "version 3"}], sources)

This fixture validates source identity and quote presence. It does not determine semantic entailment or replace labelled answer evaluation. Report abstentions and unresolved claims alongside answer scores.

Reference documentation: developers.llamaindex.ai — /python/framework/understanding/rag.

Replay a missing evidence chain before judging the answer

A multi-hop question can need two documents even when the final answer is short. Freeze both source revisions and the required spans before testing retrieval. In the owned example, catalog:v2 maps item X to group G; region:v1 maps G to Region B. A correct-looking answer without the second span still has incomplete evidence.

def chain_record(required_ids, retrieved_ids, answer, reference_answer):
    required, retrieved = set(required_ids), set(retrieved_ids)
    if not required:
        raise ValueError("A nonempty reviewed evidence reference is required")
    missing = sorted(required - retrieved)
    return {
        "required_span_coverage": len(required & retrieved) / len(required),
        "missing_span_ids": missing,
        "chain_complete": not missing,
        "answer_exact_match": answer == reference_answer,
        "answered_without_complete_chain": bool(answer) and bool(missing),
        "abstained": not bool(answer),
    }

required = ["catalog:v2", "region:v1"]
print(chain_record(required, required, "Region B", "Region B"))
print(chain_record(required, ["catalog:v2"], "Region B", "Region B"))
print(chain_record(required, required, "Region A", "Region B"))
print(chain_record(required, [], "", "Region B"))

The local replay covers a complete chain, a missing hop, a wrong answer and an abstention. An empty reference is rejected. Span coverage and answer exact match are separate outputs. The function assumes reviewed span IDs; it does not infer relevance, prove entailment or evaluate a live model. Keep contradictory source revisions as separate review cases instead of combining them silently.

For a larger research benchmark, see the MultiHop-RAG paper and its dataset repository. Their benchmark is separate from this synthetic exercise. Use claim-to-source support review when deciding whether the retrieved passages justify the answer.