A bounded Python thread pool can handle independent extraction I/O while keeping per-source failures visible. Cap the worker count, set a timeout inside the actual HTTP read and retain an error record for every failed URL. The executor alone does not impose a network timeout.
A deterministic partial-failure fixture
from concurrent.futures import ThreadPoolExecutor, as_completed
def bounded_reads(urls, read, workers=3):
outputs, errors = {}, {}
with ThreadPoolExecutor(max_workers=workers) as pool:
tasks = {pool.submit(read, url): url for url in dict.fromkeys(urls)}
for task in as_completed(tasks):
url = tasks[task]
try:
outputs[url] = task.result()
except Exception as exc:
errors[url] = type(exc).__name__
return outputs, errors
def read_fixture(url):
if url == "bad":
raise TimeoutError()
return url.upper()
outputs, errors = bounded_reads(["ok", "ok", "bad"], read_fixture, workers=3)
assert outputs == {"ok": "OK"}
assert errors == {"bad": "TimeoutError"}The local case deduplicates one repeated input and preserves one failed task. It does not establish a speedup for CPU work or any commercial site. Choose a worker count within account capacity and local resource limits.
Give the HTTP read its own deadline
This page implements the thread-pool boundary. The search-then-read workflow chooses which sources should be scheduled, and the capacity guide sizes the account workload. Keeping those tasks separate avoids repeating one broad parallel-search tutorial.
- Use a documented API request and validate the HTTP and business status.
- Set a per-request timeout and a caller-level work budget.
- Preserve URL and error category without exposing response secrets.
- Do not retry authentication or configuration failures blindly.
- Keep partial output explicitly partial when the task needs all sources.
Reference documentation: docs.python.org — /3/library/concurrent.futures.html; www.searchcans.com — /reader-api.
