Evaluate a cross-encoder reranker on a fixed candidate set. If the initial retriever changes between comparisons, a better final rank may come from different candidates rather than reranking. Keep query labels, candidate identities and the reranker version with the result.
Separate retrieval coverage from reranking order
| Question | Measurement |
|---|---|
| Was the required passage retrieved at all? | Candidate-set coverage |
| Did reranking move it into final context? | Labelled rank before and after |
| Was the important qualification retained? | Source-span support review |
| What did the reranker cost? | Observed duration and configured resource use |
| Can the run be repeated? | Model/version/license and input record |
Define the evaluation before loading a model
Create fixed query/candidate pairs and label the passages that answer each question. Include misleading near-matches and passages that share terms without supporting the answer. Retain the complete candidate order before and after the reranker.
Use the selected model’s current documentation and license. A cross-encoder score is a model output, not proof that a factual claim is correct. Check whether the selected passage actually supports the answer, including its date and limitations.
Report what was and was not tested
This revision specifies an evaluation procedure. Run it on the actual candidate set before publishing a measured improvement claim.
- Keep the candidate set unchanged in the comparison.
- Report cases where the required evidence was never retrieved.
- Preserve unsupported or contradicted final answers for review.
- Do not claim a universal accuracy improvement from a model description.
Reference documentation: sbert.net — /docs/cross_encoder/usage/usage.html.
