Evaluate fresh-web RAG against a frozen evidence set before comparing answers. A live source can change between runs, so a higher score may reflect different evidence rather than a better retrieval or generation method. Retain source revisions, question labels and supported claims together.
Freeze the parts needed to reproduce a run
| Artifact | Reason |
|---|---|
| Question set and expected support | Define the task being evaluated |
| Source URLs, dates and content hashes | Identify the actual evidence |
| Retrieved candidates and final context | Separate retrieval from generation errors |
| Model/configuration record | Identify the answer-generation setup |
| Claim/source review | Separate supported, contradicted and unresolved statements |
Keep support checks distinct from evaluator opinions
An automatic judge can help prioritize review, but its score is not factual truth. Start with source-linked claim checks and retain the exact passage being used. A quote existing in a document is a useful mechanical check; whether it supports the claim requires context and, for ambiguous cases, human review.
Do not silently refresh the source corpus halfway through a model comparison. If freshness is the variable being tested, run a separate dated experiment and state which source versions each run used.
A minimal local citation gate
def supported_answer(claims, source_map):
return all(c.get('source_id') in source_map and bool(c.get('quote'))
and c['quote'] in source_map[c['source_id']] for c in claims) and bool(claims)
sources = {"d1": "The fixture supports version 2."}
assert supported_answer([{"source_id": "d1", "quote": "version 2"}], sources)
assert not supported_answer([{"source_id": "d1", "quote": "version 3"}], sources)This fixture validates source identity and quote presence. It does not determine semantic entailment or replace labelled answer evaluation. Report abstentions and unresolved claims alongside answer scores.
Reference documentation: developers.llamaindex.ai — /python/framework/understanding/rag.
