To diagnose RAG latency, time each stage with the same monotonic clock and retain the dependencies between stages. Separate external search, page reading, vector retrieval, reranking and LLM first-token time. Parallel stage durations cannot simply be added to obtain user-visible elapsed time.
Start from one trace, not an ideal latency target
Record a request ID, stage start/end, outcome, attempts, input version and cache-hit decision. LLM first-token time ends when the first token is observed, not when generation finishes. A failed retrieval should remain in the trace. Do not compare a warm-cache success with an uncached request without labeling the difference.
Calculate wall time with parallel reads
from math import isfinite
def retrieval_trace(spans):
if not spans:
raise ValueError("Trace is empty")
for name, start, end in spans:
if not isinstance(start, (int,float)) or not isinstance(end, (int,float)) or not isfinite(start) or not isfinite(end) or end < start:
raise ValueError("Invalid span: " + name)
if len({s[0] for s in spans}) != len(spans):
raise ValueError("Stage names must be unique")
start, end = min(s[1] for s in spans), max(s[2] for s in spans)
return {"wall_clock_ms":end-start,
"stage_ms":{name:b-a for name,a,b in spans},
"summed_stage_ms":sum(b-a for _,a,b in spans)}
if __name__ == "__main__":
# Fictional elapsed timestamps from one clock, not measured service latency.
spans = [("search",0,20),("read_A",20,50),("read_B",20,60),
("rerank",60,70),("llm_first_token",70,100)]
print(retrieval_trace(spans))
The synthetic trace takes one hundred milliseconds from its first start to last end. Stage durations sum to one hundred and thirty because the two reads overlap. Those values illustrate the arithmetic; they do not establish a target or measured SearchCans performance. An end before a start is rejected. The timestamps must come from one clock domain.
Locate the dependency that delays the answer
After search completes, reads may execute in parallel subject to the account and application limits. If the answer needs both sources, the slower required read delays reranking. If one optional read can be omitted, make that a documented evidence policy rather than silently dropping it to improve a latency number. Vector retrieval may be independent or depend on newly extracted text; record the actual graph.
A cache can reduce repeated work, but its age and invalidation policy affect freshness. Record the source version and decide when the task requires revalidation. For an experiment, hold the question set, market, cache condition and evidence acceptance rules constant, then compare per-request traces and failure rates. No universal “ideal milliseconds” follows from this method.
Use the external search benchmark for source freshness and the real-time RAG architecture for pipeline design. This page owns timing attribution rather than duplicating the ingestion architecture.
