Before tuning a hybrid RAG system, test how its rank-fusion step handles repeated document IDs and complementary candidates. Keep document identity stable across the keyword and vector lists. A score merge that double-counts a duplicate can change the final ordering without adding new evidence.
Fuse ranks rather than incomparable raw scores
from collections import defaultdict
def rrf(rank_lists, k=60):
scores = defaultdict(float)
for ranking in rank_lists:
for rank, doc_id in enumerate(dict.fromkeys(ranking), 1):
scores[doc_id] += 1 / (k + rank)
return sorted(scores, key=lambda key: (-scores[key], key))
assert rrf([["a", "b", "b"], ["b", "c"]]) == ["b", "a", "c"]This RRF fixture deduplicates each input list before assigning ranks, then sums rank contributions across lists. It uses a fixed k of 60 as an example parameter, not a universal optimum. The repeated b in one list does not receive an extra vote.
Build failure cases around document identity
| Case | Assertion |
|---|---|
| Repeated ID in one ranking | Count that list’s document once |
| Same document in two rankings | Keep both lists’ rank contributions |
| Complementary document | Do not discard it just because it appears in one list |
| Equal fusion score | Use a deterministic tie rule |
| Different IDs for a copied passage | Resolve identity before interpreting coverage |
Evaluate retrieval after the contract test
The fixture checks ordering and deduplication. It does not show that hybrid retrieval improves model answers. For a quality comparison, freeze questions and relevant document labels, retain the same candidate budget, and measure whether the fused set contains the required supporting passages.
Use the parent hybrid-search guide for the production architecture and tuning workflow. This page isolates fusion correctness and its failure cases so those issues can be tested before changing the rest of the pipeline.
Reference documentation: www.elastic.co — /docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion.
