Evaluate Parallel Retrieval: Coverage, Duplicates and Unsupported Answers

Evaluate parallel retrieval against the same questions and evidence requirements used for the sequential run. Concurrency changes scheduling; it does not automatically make sources more independent or answers more accurate. Report coverage, duplicates, failures and supported answers separately.

Hold the research task fixed

Measure evidence before celebrating duration

Retain unique supporting sources, copied passages, unavailable reads and unresolved claims for each run. If one run obtains more independent supporting evidence, identify which selection or scheduling behavior caused that difference. Do not treat duplicated URLs as improved coverage.

A bounded parallel run can return partial output. Report that outcome even if its duration is shorter. Compare duration only alongside the work completed and the evidence accepted.

Keep the conclusions within the experiment

Execution and partial-failure checks belong to the linked scheduling guide; this page evaluates evidence selection. This page supplies a controlled evaluation design; it does not report an unperformed accuracy benchmark.

• Use dated run logs and fixed input manifests.
• Report incomplete and unsupported cases, not only successful answers.
• Treat a faster local fixture as scheduling evidence, not a whole-provider benchmark.
• Distinguish the general technique from the Parallel.ai provider brand and SearchCans account lanes.

Reference documentation: www.searchcans.com — /google-search-api; www.searchcans.com — /reader-api.

Calculate relevance and duplication on a labelled candidate pool

Before running a comparison, freeze the question, the permitted source snapshots and the human relevance labels. The function below deduplicates exact URLs after removing fragments. It does not merge mirrors or different URLs containing copied text; review those separately.

from urllib.parse import urldefrag

def pool_metrics(returned_urls, relevant_pool_urls, k=3):
    if k < 1 or not relevant_pool_urls:
        raise ValueError("Positive k and a nonempty labelled relevant pool required")
    unique = list(dict.fromkeys(urldefrag(u)[0] for u in returned_urls))
    relevant = {urldefrag(u)[0] for u in relevant_pool_urls}
    top = unique[:k]
    hits = sum(u in relevant for u in top)
    first = next((i for i, u in enumerate(unique, 1) if u in relevant), None)
    return {"unique_urls": unique,
            "duplicate_fraction": (len(returned_urls) - len(unique)) / len(returned_urls) if returned_urls else 0,
            "precision_returned_top_k": hits / len(top) if top else 0,
            "labelled_pool_recall_at_k": hits / len(relevant),
            "reciprocal_rank": 1 / first if first else 0,
            "returned_top_k_count": len(top), "labelled_relevant_pool_count": len(relevant)}

returned = ["https://example.test/no", "https://example.test/a",
            "https://example.test/a#copy", "https://example.test/b"]
relevant = {"https://example.test/a", "https://example.test/b"}
print(pool_metrics(returned, relevant, k=3))

The synthetic run has one duplicate among four returned URLs. After deduplication, two of three returned candidates are labelled relevant, reciprocal rank is 0.5 and recall against the two labelled relevant pool items is 1. The precision denominator is the number of actually returned unique items up to k. Report that count so a short result list is visible. Pool recall is not recall against the unknown whole web.

Labels come from a reviewer; this code does not determine whether a page supports the answer. Run the same labelled pool through each configuration and retain the original order. Use the official retrieval-evaluation reference for broader metric implementations and RAG evaluation to test the answer separately. Local scheduling belongs in the partial-failure execution guide, not in an accuracy claim.