Self-Correcting RAG with SearchCans: CRAG Guide

Standard RAG (Retrieval-Augmented Generation) has a fatal flaw: It blindly trusts the retrieval.

If your vector database returns outdated documents, or worse, irrelevant ones, your LLM will hallucinate a confident answer based on bad data.

Enter Corrective RAG (CRAG) and Self-RAG.

These advanced architectures introduce a “Self-Correction” loop. The system evaluates the quality of retrieved documents. If they are deemed “Ambiguous” or “Incorrect,” the agent automatically triggers a web search to find the truth.

In this guide, we will build the “Web Search Node” for a CRAG pipeline using SearchCans.

Key Takeaways

• Corrective RAG (CRAG) eliminates static RAG hallucinations by routing “incorrect” or “ambiguous” retrievals to a live web search , grounding agents in current facts before generating an answer

• SearchCans’ SERP + Reader API combo provides the production-grade “real-time lifeline” for self-correcting agents at $0.56/1K requests, far below $15+ alternatives

• The Reader API ("mode": 1) returns full LLM-ready Markdown from source URLs , giving CRAG agents complete page context, not just snippets, to correct hallucinations

• LangGraph integration is straightforward: CRAG’s web_search node wraps a SearchCansRetriever that handles SERP lookup and deep URL reading in a single pipeline step

• Up to 113 Parallel Lanes mean CRAG loops that retry 10+ times per query won’t bottleneck on rate limits

The CRAG Architecture: Traffic Lights for Data

Conceptually, CRAG acts like a traffic light for your data retrieval:

1. Green (Correct): The vector DB documents are relevant. Generate answer.

1. Red (Incorrect): The documents are wrong. Discard them and search the web.

1. Amber (Ambiguous): The documents are vague. Combine them with a web search for clarity.

The Role of SearchCans

In the “Red” and “Amber” states, your agent needs to leave the internal database and check the open internet.

For a production agent that might loop and retry 10 times per query, you need an API that is:

Cost-Effective

SearchCans is $0.56/1k (vs $15+ for others).

Deeply Grounded

We don’t just return snippets. Our Reader API fetches the full markdown of the source page.

Implementation: The “Web Search” Node

Let’s implement the search component for a LangGraph workflow.

import requests

class SearchCansRetriever:
   """
   A robust Web Search Node for CRAG/Self-RAG pipelines.
   """
   def __init__(self, api_key):
       self.api_key = api_key
       self.search_endpoint = "https://www.searchcans.com/api/v1/search"
       self.reader_endpoint = "https://www.searchcans.com/api/v1/url"
       self.headers = {"Authorization": f"Bearer {self.api_key}"}

   def corrective_search(self, query: str):
       print(f"CRAG Triggered: Searching web for '{query}'...")

       # Step 1: Search Google for fresh links
       search_params = {
           "q": query,
           "engine": "google",
           "num": 3  # Get top 3 candidates
       }

       try:
           resp = requests.get(self.search_endpoint, headers=self.headers, params=search_params)
           results = resp.json().get("organic_results", [])

           if not results:
               return "No relevant web results found."

           # Step 2: "Deep Read" the top result to ground the answer
           top_link = results[0]['link']
           return self._read_content(top_link)

       except Exception as e:
           return f"Search failed: {str(e)}"

   def _read_content(self, url):
       print(f"Reading source: {url}...")
       # Use Reader API to get Clean Markdown
       payload = {
           "s": url,
           "t": "url",
           "mode": 1,    # Use headless browser for dynamic sites
           "w": 2000     # Wait for content to hydrate
       }

       resp = requests.post(self.reader_endpoint, headers=self.headers, json=payload)
       data = resp.json()

       # Prefer Markdown for LLM context window efficiency
       content = data.get("data", {}).get("markdown", "") or ""
       return f"WEB CONTEXT FROM {url}:\n{content[:5000]}"

Integrating with LangGraph

In a standard LangGraph setup, you would add this as a node. When the “Grader” node determines that internal documents are insufficient, it routes the state to this web_search node.

# Pseudo-code for LangGraph integration
def web_search_node(state):
   question = state["question"]
   search_tool = SearchCansRetriever(api_key="YOUR_KEY")

   # Perform corrective search
   web_context = search_tool.corrective_search(question)

   # Update state with new, grounded knowledge
   return {"documents": [web_context], "question": question}

# Add to graph
workflow.add_node("web_search", web_search_node)

Building a Document Grader

The grader evaluates document relevance using an LLM:

from openai import OpenAI

client = OpenAI(api_key="YOUR_OPENAI_KEY")

def grade_documents(question: str, documents: list) -> str:
   """
   Grades retrieved documents for relevance.
   Returns: 'correct', 'incorrect', or 'ambiguous'
   """
   doc_text = "\n\n".join(documents)

   prompt = f"""
   Question: {question}

   Retrieved Documents:
   {doc_text}

   Are these documents relevant and sufficient to answer the question?
   Respond with ONLY: 'correct', 'incorrect', or 'ambiguous'
   """

   response = client.chat.completions.create(
       model="gpt-4o",
       messages=[{"role": "user", "content": prompt}]
   )

   return response.choices[0].message.content.strip().lower()

Complete CRAG Flow

Here’s how the complete system works:

def crag_pipeline(question: str):
   # Step 1: Retrieve from vector DB
   vector_docs = vector_db.retrieve(question)

   # Step 2: Grade the documents
   grade = grade_documents(question, vector_docs)

   # Step 3: Route based on grade
   if grade == "correct":
       context = "\n\n".join(vector_docs)
   elif grade == "incorrect":
       # Discard vector results, search web
       searcher = SearchCansRetriever(api_key="YOUR_KEY")
       context = searcher.corrective_search(question)
   else:  # ambiguous
       # Combine both sources
       searcher = SearchCansRetriever(api_key="YOUR_KEY")
       web_context = searcher.corrective_search(question)
       context = f"Internal:\n{vector_docs[0]}\n\nWeb:\n{web_context}"

   # Step 4: Generate final answer
   return generate_answer(question, context)

Why This Matters for “Self-Correction”

Self-RAG agents are designed to be autonomous. They critique their own outputs and iterate.

If your agent decides its answer is hallucinated, it must have a reliable way to get fresh data.

Without SearchCans

The agent is stuck with its internal training data (which caused the hallucination).

With SearchCans

The agent has a real-time “lifeline” to verify facts against the live internet.

Conclusion

Building a “Self-Correcting” agent is the hallmark of a senior AI engineer. It moves your system from a fun toy to a reliable enterprise tool.

By integrating SearchCans, you provide the affordable, high-speed infrastructure needed to support the multiple search-and-verify loops that CRAG architectures require.

Rule G+: The “Not For” Clause: SearchCans is purpose-built for real-time data acquisition and LLM-ready content extraction. It is NOT a persistent vector store, a full-browser automation testing framework, or a replacement for your RAG retrieval layer. Use it as the “web search” fallback node in CRAG , when internal documents fail the grader, SearchCans provides the fresh factual context needed to self-correct.

Frequently Asked Questions

Q: What is Corrective RAG (CRAG) and how does it differ from standard RAG?

A: Standard RAG blindly trusts retrieved documents even when they are outdated or irrelevant, leading to confident hallucinations. CRAG adds a grader step that evaluates document quality , routing “incorrect” queries to a live web search and “ambiguous” queries to a hybrid of internal + web data before generating an answer.

Q: Why use SearchCans as the web search node in a CRAG pipeline?

A: CRAG agents can loop and retry 10+ times per query, making cost per search critical. SearchCans costs $0.56/1K requests versus $15+ for alternatives, and its Reader API returns full LLM-ready Markdown from source URLs , giving agents complete page context to correct hallucinations, not just short snippets.

Q: What is the difference between the SERP API and Reader API roles in CRAG?

A: The SERP API (/api/search) finds the top candidate URLs for a corrective query. The Reader API (/api/url) then deep-reads the top result, returning clean Markdown. This two-step pattern grounds the LLM in a full source document rather than a brief snippet.

Q: How many correction loops can SearchCans support without hitting rate limits?

A: SearchCans uses a Parallel Lanes model with no hourly limits. Up to 113 simultaneous lanes mean CRAG pipelines can run dozens of correction loops concurrently across multiple queries without queuing delays.

Q: When should I combine CRAG with a vector database?

A: CRAG is most valuable when your vector database contains time-sensitive documents that may become stale. For purely static knowledge, standard RAG may suffice. Add CRAG when accuracy requirements demand real-time fact verification against the live web.

Resources

Related Topics:

• AI Agent Internet Access Architecture – Architecture guide for connected agents

• URL to Markdown API Benchmark – Why clean data matters for RAG

• Adaptive RAG Router – Dynamic knowledge routing

• Deep Research Agent – Advanced LangGraph patterns

• Hybrid RAG Tutorial – Combining vector and web search

Get Started:

• Free Trial – Get 100 free credits

• API Documentation – Technical reference

• Pricing – Transparent costs

• Playground – Test in browser

SearchCans provides real-time data for AI agents. Start building now →