Multiple search agents can produce more candidates without producing more relevant evidence. Evaluate the combined result set after deduplication, with judgments recorded against a specific question. Keep the single-agent baseline, query budget and decision rules visible. Parallel execution and improved relevance are different claims.
Define the experiment before running it
Choose questions from the task you serve. For each question, define what counts as a relevant source and what information would be missing from an acceptable answer. Save the exact agent instructions, rewritten queries, market, tool versions and request budget. Use the same allowed sources and budget policy for the baseline and proposed workflow.
Score candidates with recorded judgments
from urllib.parse import urlsplit, urlunsplit
def canonical_candidate(url):
p = urlsplit(url)
if p.scheme not in ("http", "https") or not p.hostname:
raise ValueError("Absolute HTTP URL required")
# Query/path are preserved; only the fragment is removed.
return urlunsplit((p.scheme.lower(), p.netloc.lower(), p.path or "/", p.query, ""))
def evaluate_candidates(agent_urls, judged_relevant, credited_calls):
urls = list(dict.fromkeys(canonical_candidate(u) for lane in agent_urls for u in lane))
if any(u not in judged_relevant for u in urls):
raise ValueError("Every candidate needs a recorded relevance judgment")
if any(type(judged_relevant[u]) is not bool for u in urls):
raise ValueError("Use explicit binary relevance labels")
if credited_calls < 0:
raise ValueError("Credit count cannot be negative")
hits = sum(judged_relevant[u] for u in urls)
return {"unique_urls":len(urls), "relevant_urls":hits,
"precision":hits/len(urls) if urls else None,
"credits_per_relevant_url":credited_calls/hits if hits else None}
if __name__ == "__main__":
lanes = [["https://example.com/a", "https://example.com/a#part"], ["https://example.com/b"]]
print(evaluate_candidates(lanes, {"https://example.com/a":True,"https://example.com/b":False}, 3))
The synthetic example has three returned candidates but two unique URLs after removing a fragment. One is judged relevant, so precision is one half. Three credited calls divided by one relevant URL yields three credits per relevant URL. These are fixture calculations, not observed quality gains. Unjudged URLs are rejected instead of silently treated as relevant.
Measure the failures that averaging can hide
Record redundant sources, missing subquestions, conflicting passages and invalid citations separately. Candidate precision does not prove the final answer is correct. If you also measure coverage, define a judged reference set and publish how it was assembled; do not use a larger URL count as coverage. If costs include Reader or model calls, record those separately from search credits.
A fair comparison repeats the same question set and exposes raw per-question results, including failures. Do not report a universal percentage gain from a small fixture. For orchestration and permission boundaries, use the agent architecture. For timing, use the search benchmark contract.
