An AI news aggregator works best as a two-stage pipeline: use a SERP API to discover candidate URLs, then use a Reader API to extract the few pages worth sending to an LLM. Deduplication, source tracking, and a clear evidence boundary matter as much as the model prompt.
The workflow at a glance
- Turn a user’s topics into a small set of search queries.
- Discover candidate results with the SERP API.
- Normalize URLs and remove duplicates.
- Select pages using recency, source, and relevance rules.
- Extract selected pages as Markdown with the Reader API.
- Summarize with citations that point back to the source URLs.
For deeper Reader implementation details, see SearchCans Reader API for RAG content parsing.
Stage 1: Discover candidate stories
The current SearchCans endpoint is POST https://www.searchcans.com/api/v1/search. A query uses t: "google" and s for the search text. Country, language, page, and optional result features can be added when the application needs them.
import requests
response = requests.post(
"https://www.searchcans.com/api/v1/search",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"t": "google",
"s": "AI agent infrastructure updates",
"country": "us",
"language": "en",
"p": 1,
"d": 30000,
},
timeout=35,
)
response.raise_for_status()
organic = response.json()["data"]["organic"]Do not send every result to the next stage. Keep the result URL, title, snippet, query, position, and retrieval timestamp. That metadata lets the curation layer explain why a story was selected.
Stage 2: Deduplicate before extraction
Normalize obvious URL differences such as tracking parameters, fragments, and trailing slashes where your policy permits. Keep a canonical URL field and a source URL field so the final summary can preserve the original link. Also deduplicate by article title and publication timestamp when several sites syndicate the same report.
A useful selection record looks like this:
| Field | Purpose |
|---|---|
canonical_url |
Stable page identity |
discovered_by |
Query or feed that found it |
title |
Human review and display |
source_domain |
Source diversity and trust rules |
published_at |
Recency filtering |
selection_reason |
Audit trail for the curator |
Stage 3: Read selected pages as Markdown
The Reader API endpoint is POST https://www.searchcans.com/api/v1/url. Use t: "url" and the page URL in s. Standard mode is the right starting point for ordinary pages. Use mode: 1 when the page needs headless browser rendering, and escalate the proxy tier only when the target blocks the standard path.
reader = requests.post(
"https://www.searchcans.com/api/v1/url",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={
"t": "url",
"s": "https://example.com/article",
"mode": 0,
"w": 1000,
"d": 30000,
"proxy": 0,
},
timeout=35,
)
reader.raise_for_status()
article = reader.json()["data"]
markdown = article["markdown"]Reader standard mode costs 2 credits. The current pricing and Reader documentation should be checked before changing a production budget. Treat extracted Markdown as source material, not as proof that every claim is correct.
Stage 4: Summarize with an evidence boundary
The LLM prompt should require a short summary, the source title, the source URL, the publication date when available, and a list of claims that need verification. It should not fill gaps with invented dates or numbers. If two sources disagree, preserve the disagreement or send the item to review instead of blending the claims into one confident sentence.
For a daily briefing, store the source URL next to every sentence or bullet that depends on it. This is more useful than adding a generic references section after the model has already lost track of provenance.
Cost and throughput controls
Keep discovery and extraction in separate queues. SearchCans Parallel Lanes represent simultaneous in-flight requests. The current plan pages list 2 lanes for Standard, 5 for Starter, 37 for Pro, and 113 for Ultimate. Set worker limits below the available capacity, cache stable extraction results in your own application where allowed, and do not extract the same URL for every subscriber.
Frequently Asked Questions
Q: Why not send raw SERP results directly to the LLM?
A: SERP results are useful for discovery but usually do not contain enough page context. Reader turns selected URLs into cleaner Markdown so the model receives the article content together with source metadata.
Q: Should every discovered URL go through Reader?
A: No. Filter, deduplicate, and rank first. Extraction is a separate cost and latency step.
Q: How do I handle JavaScript-heavy news pages?
A: Retry the selected URL with Reader mode: 1, then consider a proxy tier only if the standard network path fails. Record which mode and tier produced the evidence.
Conclusion
The SERP and Reader combination is valuable because each API has a focused job. Search discovers candidates, Reader prepares selected pages, and the curation layer preserves source identity. That division produces briefings that are easier to scale, audit, and trust.
