LLM Agent Retrieval Latency: Profile the Critical Path

Reducing LLM agent retrieval latency starts with the critical path. Time discovery, source reads and later model work separately, then identify which stages can overlap. Summing request durations is not the same as measuring wall time. A faster pipeline is only useful when required evidence still arrives and failed branches remain visible.

Separate a product name from an execution pattern

Queries about Basic or Advanced search modes can refer to Parallel.ai’s product. Those names are documented in Parallel’s search modes; they are not SearchCans mode names. This guide measures a general discovery-and-read pipeline and does not present a provider mode switch as a SearchCans feature.

Record the dependencies before changing concurrency

Independent reads may overlap. A read that needs the previous page’s output is a dependency and belongs in a sequential path. Use bounded asyncio scheduling to enforce task limits and cancellation behavior. Available Parallel Lanes describe in-flight capacity, not a promise of a particular elapsed time.

Instrument a bounded retrieval slice

The example profiles one discovery callback and up to three independent source callbacks. Set the bound no higher than available account capacity. The local demonstration uses two delayed synthetic reads and no network credentials. Production callbacks need their own bounded HTTP timeouts; this profiler does not cancel remote jobs.

import asyncio
from time import perf_counter

async def profile_pipeline(discover, read, max_sources=3):
    if not 1 <= max_sources <= 3:
        raise ValueError("Set a bounded source count from account capacity")
    events = []
    started = perf_counter()
    async def timed(label, call):
        begin = perf_counter()
        try:
            value = await call()
            status = "ok"
        except Exception as error:
            value = {"error_type": type(error).__name__}
            status = "failed"
        end = perf_counter()
        events.append({"phase": label, "status": status,
                       "start_seconds": begin - started,
                       "end_seconds": end - started,
                       "elapsed_seconds": end - begin})
        return status, value
    state, urls = await timed("discover", discover)
    if state != "ok" or not isinstance(urls, list):
        return {"status": "discovery_failed", "events": events}
    if any(not isinstance(url, str) or not url.startswith("https://") for url in urls):
        raise ValueError("Discovery callback must return approved HTTPS URLs")
    urls = list(dict.fromkeys(urls))[:max_sources]
    if not urls:
        return {"status": "no_sources", "events": events, "outputs": []}
    async def checked_read(url):
        value = await read(url)
        if not isinstance(value, dict) or not isinstance(value.get("text"), str) or not value["text"].strip():
            raise ValueError("Read callback must return useful text")
        return value
    read_started = perf_counter()
    outputs = await asyncio.gather(*[
        timed("read:" + url, lambda url=url: checked_read(url)) for url in urls])
    read_wall = perf_counter() - read_started
    reader_sum = sum(e["elapsed_seconds"] for e in events if e["phase"].startswith("read:"))
    return {"status": "partial" if any(s != "ok" for s, v in outputs) else "ok",
            "events": events, "outputs": outputs,
            "read_wall_seconds": read_wall, "sum_reader_seconds": reader_sum,
            "pipeline_wall_seconds": perf_counter() - started,
            "missing": [u for u, (s, v) in zip(urls, outputs) if s != "ok"]}

if __name__ == "__main__":
    import json
    async def discover():
        return ["https://example.test/a", "https://example.test/b"]
    async def read(url):
        await asyncio.sleep(0.02)
        return {"url": url, "text": "Synthetic source"}
    print(json.dumps(asyncio.run(profile_pipeline(discover, read, 2)), indent=2))

Interpret the trace before claiming an improvement

sum_reader_seconds is the sum of observed read durations. read_wall_seconds is the duration of the concurrent read stage. Their difference reflects overlap and scheduling in this run, not a guaranteed speedup. pipeline_wall_seconds includes discovery and the read stage; it excludes any model synthesis not supplied by the callbacks.

If discovery fails, do not report an empty successful pipeline. If one source fails, inspect whether the remaining evidence can answer the task. An early answer that silently drops a necessary branch is a completeness failure. Run comparable repetitions with the same inputs and cache conditions before describing a latency distribution.

Keep result relevance evaluation separate from timing. Use the API benchmark method for a repeatable latency study. This page owns stage-level diagnosis, rather than a second generic concurrency or provider-pricing tutorial.

Sources and verification

• Python: coroutine scheduling and gather
• Parallel.ai: its own product modes