Build a Mini-DeepResearch Agent with SearchCans

Let’s build a functional DeepResearch agent from scratch. By the end of this tutorial, you’ll have a working system that can autonomously research any topic and generate comprehensive reports.

What We’re Building

A Mini-DeepResearch agent that:

• Takes a research question as input

• Searches the web using SERP API

• Extracts content using Reader API

• Synthesizes findings with GPT-4

• Generates a cited research report

Time to build: 30 minutes Experience level: Intermediate Python

How the Workflow Fits Together

The agent uses a search-then-read sequence. The SERP request is responsible for discovery: it returns candidate pages, titles, snippets, and URLs for the research question. The Reader request is responsible for evidence: it turns selected URLs into cleaner content that can be quoted, summarized, and checked against the source page.

That separation matters because a search snippet is not a complete source. A snippet can help rank candidates, but it may omit qualifications, dates, tables, or context. The agent should therefore keep the source URL beside every extracted passage and should treat the final report as a synthesis of cited evidence, not as a replacement for the sources.

For a dependable run, add a few practical checks around the sample code. Validate that the API key exists before making a request, handle non-2xx responses, and record the URL and response status for each source. Set a request timeout and use bounded retries for temporary failures. A failed extraction should be recorded and skipped or retried, rather than silently becoming an empty source.

The report prompt should also distinguish facts from interpretation. Ask the model to return the source URL, the supporting passage, and any uncertainty for each claim. When sources disagree, preserve the disagreement and explain which evidence is newer or more directly relevant. This makes the output easier to audit and reduces the risk of a polished report hiding weak evidence.

Finally, control cost and freshness deliberately. Start with a small number of search results, extract only the pages that answer the question, and cap the amount of text sent to the model. For recurring research, store the fetch time and a content hash so that unchanged pages do not need to be processed again. SearchCans plan limits, credit costs, and API parameters can change, so verify the current documentation before moving this tutorial into production.

Prerequisites

# Required accounts (all have free tiers)
1. SearchCans account (SERP + Reader API)
2. OpenAI account (GPT-4 API)

# Python packages
pip install requests openai python-dotenv

Get API Keys:

• SearchCans: Sign up free

• OpenAI: https://platform.openai.com/

Step 1: Project Setup

Create project structure:

mkdir deepresearch-agent
cd deepresearch-agent
touch research_agent.py .env

Create .env file:

SEARCHCANS_API_KEY=your_searchcans_key_here
OPENAI_API_KEY=your_openai_key_here

Step 2: Core Research Agent Class

# research_agent.py

import os
import requests
from openai import OpenAI
from dotenv import load_dotenv
from typing import List, Dict

load_dotenv()

class MiniDeepResearch:
   def __init__(self):
       self.api_key = os.getenv("SEARCHCANS_API_KEY")
       self.serp_url = "https://www.searchcans.com/api/v1/search"
       self.reader_url = "https://www.searchcans.com/api/v1/url"
       self.client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

   def search(self, query: str, num_results: int = 10) -> List[Dict]:
       """Search the web using SERP API"""
       response = requests.post(
           self.serp_url,
           headers={"Authorization": f"Bearer {self.api_key}"},
           json={
               "s": query,
               "t": "google",
               "num": num_results
           },
           timeout=30
       )
       response.raise_for_status()
       return response.json().get("data", [])

   def extract_content(self, url: str) -> Dict:
       """Extract clean content from URL using Reader API"""
       try:
           response = requests.post(
               self.reader_url,
               headers={"Authorization": f"Bearer {self.api_key}"},
               json={"s": url, "t": "url", "mode": 1, "w": 2000},
               timeout=35
           )
           response.raise_for_status()
           return response.json().get("data", {})
       except Exception as e:
           print(f"Failed to extract {url}: {e}")
           return None

   def synthesize(self, prompt: str) -> str:
       """Use GPT-4 to synthesize information"""
       response = self.client.chat.completions.create(
           model="gpt-4",
           messages=[{"role": "user", "content": prompt}],
           temperature=0.3
       )
       return response.choices[0].message.content

Step 3: Multi-Step Research Logic

Add research orchestration:

class MiniDeepResearch:
   # ... (previous methods)

   def research(self, question: str, max_sources: int = 8) -> str:
       """
       Main research method - orchestrates the full research process
       """
       print(f"\n🔍 Researching: {question}\n")

       # Step 1: Initial search
       print("Step 1: Searching web...")
       search_results = self.search(question, num_results=max_sources)
       print(f"Found {len(search_results)} results\n")

       # Step 2: Extract content from top results
       print("Step 2: Extracting content...")
       extracted_contents = []
       for i, result in enumerate(search_results[:max_sources], 1):
           print(f"  [{i}/{max_sources}] {result['title'][:60]}...")

           content = self.extract_content(result['url'])
           if content and content.get('markdown'):
               extracted_contents.append({
                   "url": result['url'],
                   "title": result['title'],
                   "domain": result.get('domain', ''),
                   "content": content['markdown'][:3000]  # First 3000 chars
               })

       print(f"Successfully extracted {len(extracted_contents)} sources\n")

       # Step 3: Analyze each source
       print("Step 3: Analyzing sources...")
       analyses = []
       for i, source in enumerate(extracted_contents, 1):
           print(f"  [{i}/{len(extracted_contents)}] Analyzing {source['domain']}...")

           # See our guide on advanced prompt engineering for AI agents: /blog/advanced-prompt-engineering-for-ai-agents/
           analysis_prompt = f"""
Analyze this content in relation to the question: "{question}"

Content from {source['domain']}:
{source['content']}

Extract:
1. Key facts and data relevant to the question
2. Important quotes or insights
3. Any contradictions or uncertainties

Format as: [Key Facts] ... [Quotes] ... [Notes] ...
"""

           analysis = self.synthesize(analysis_prompt)
           analyses.append({
               "source": source,
               "analysis": analysis
           })

       # Step 4: Synthesize final report
       print("\nStep 4: Synthesizing final report...")

       # Prepare context for final synthesis
       context = "\n\n---\n\n".join([
           f"SOURCE: {a['source']['title']} ({a['source']['url']})\n{a['analysis']}"
           for a in analyses
       ])

       final_prompt = f"""
You are a research analyst. Based on the following analyses from {len(analyses)} sources,
create a comprehensive research report answering: "{question}"

SOURCES AND ANALYSES:
{context}

Create a report with:
1. Executive Summary (2-3 paragraphs)
2. Key Findings (bullet points with citations)
3. Detailed Analysis (organized by themes)
4. Conclusion
5. Sources (numbered list)

Use citations like [1], [2], etc. to reference sources.
Be objective and note any conflicting information.
"""

       final_report = self.synthesize(final_prompt)

       # Append sources
       sources_list = "\n".join([
           f"[{i+1}] {a['source']['title']} - {a['source']['url']}"
           for i, a in enumerate(analyses)
       ])

       full_report = f"{final_report}\n\n## Sources\n\n{sources_list}"

       print("Research complete!\n")
       return full_report

Step 4: Enhanced Features

Add follow-up questions and multi-query research:

class MiniDeepResearch:
   # ... (previous methods)

   def generate_sub_questions(self, main_question: str) -> List[str]:
       """Generate sub-questions to research comprehensively"""
       prompt = f"""
Given this research question: "{main_question}"

Generate 3-5 specific sub-questions that would help answer it comprehensively.

Return only the questions, one per line, without numbering.
"""

       response = self.synthesize(prompt)
       sub_questions = [q.strip() for q in response.strip().split('\n') if q.strip()]
       return sub_questions[:5]

   def deep_research(self, question: str) -> str:
       """
       Enhanced research with sub-questions
       """
       print(f"\n🔬 Deep Research Mode: {question}\n")

       # Generate sub-questions
       print("Generating sub-questions...")
       sub_questions = self.generate_sub_questions(question)
       print(f"Sub-questions:\n")
       for i, sq in enumerate(sub_questions, 1):
           print(f"  {i}. {sq}")
       print()

       # Research each sub-question
       all_findings = []

       # Main question
       main_findings = self.research(question, max_sources=5)
       all_findings.append({
           "question": question,
           "findings": main_findings
       })

       # Sub-questions
       for sq in sub_questions[:3]:  # Limit to 3 to save API calls
           findings = self.research(sq, max_sources=3)
           all_findings.append({
               "question": sq,
               "findings": findings
           })

       # Synthesize everything
       print("\n📊 Creating comprehensive report...\n")

       combined_context = "\n\n".join([
           f"# Research on: {f['question']}\n\n{f['findings']}"
           for f in all_findings
       ])

       final_prompt = f"""
Based on comprehensive research covering multiple aspects, create a final report
answering: "{question}"

RESEARCH FINDINGS:
{combined_context}

Create an authoritative report that:
- Synthesizes all research
- Identifies key themes
- Resolves contradictions
- Provides actionable insights
- Includes comprehensive citations
"""

       final_report = self.synthesize(final_prompt)
       return final_report

Step 5: Usage Examples

# Basic usage
def main():
   agent = MiniDeepResearch()

   # Simple research
   report = agent.research("What is the SERP API market size in 2025?")
   print(report)

   # Save to file
   with open("research_report.md", "w") as f:
       f.write(report)

if __name__ == "__main__":
   main()

Deep research mode:

# For comprehensive analysis
agent = MiniDeepResearch()
report = agent.deep_research("Analyze AI trends in healthcare 2025")

Batch research:

# Research multiple topics
questions = [
   "AI in finance 2025 trends",
   "Best CRM software for startups",
   "Vector database comparison"
]

for q in questions:
   report = agent.research(q)
   filename = q.replace(" ", "_")[:50] + ".md"
   with open(filename, "w") as f:
       f.write(report)

Step 6: Advanced Optimizations

Parallel Processing

from concurrent.futures import ThreadPoolExecutor

class OptimizedDeepResearch(MiniDeepResearch):
   def research_parallel(self, question: str, max_sources: int = 8) -> str:
       """Research with parallel content extraction"""

       # Search
       search_results = self.search(question, num_results=max_sources)

       # Extract in parallel
       with ThreadPoolExecutor(max_workers=5) as executor:
           futures = [
               executor.submit(self.extract_content, result['url'])
               for result in search_results[:max_sources]
           ]

           extracted_contents = []
           for future, result in zip(futures, search_results[:max_sources]):
               content = future.result()
               if content and content.get('markdown'):
                   extracted_contents.append({
                       "url": result['url'],
                       "title": result['title'],
                       "content": content['markdown'][:3000]
                   })

       # Continue with synthesis...
       return self._synthesize_report(question, extracted_contents)

Speed improvement: 3-5x faster

Caching

import hashlib
import json
import os

class CachedDeepResearch(MiniDeepResearch):
   def __init__(self):
       super().__init__()
       self.cache_dir = "research_cache"
       os.makedirs(self.cache_dir, exist_ok=True)

   def _cache_key(self, text: str) -> str:
       return hashlib.md5(text.encode()).hexdigest()

   def cached_search(self, query: str, **kwargs) -> List[Dict]:
       cache_key = self._cache_key(query)
       cache_file = f"{self.cache_dir}/{cache_key}.json"

       if os.path.exists(cache_file):
           with open(cache_file, "r") as f:
               return json.load(f)

       results = self.search(query, **kwargs)

       with open(cache_file, "w") as f:
           json.dump(results, f)

       return results

Cost savings: 50-70% for repeated queries

Complete Working Example

#!/usr/bin/env python3
"""
Mini DeepResearch Agent
A simple but functional autonomous research agent
"""

import os
import requests
from openai import OpenAI
from dotenv import load_dotenv

load_dotenv()

class MiniDeepResearch:
   def __init__(self):
       self.api_key = os.getenv("SEARCHCANS_API_KEY")
       self.serp_url = "https://www.searchcans.com/api/v1/search"
       self.reader_url = "https://www.searchcans.com/api/v1/url"
       self.client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

   def search(self, query, num=10):
       response = requests.post(
           self.serp_url,
           headers={"Authorization": f"Bearer {self.api_key}"},
           json={"s": query, "t": "google", "num": num},
           timeout=30
       )
       return response.json().get("data", [])

   def extract(self, url):
       try:
           response = requests.post(
               self.reader_url,
               headers={"Authorization": f"Bearer {self.api_key}"},
               json={"s": url, "t": "url", "mode": 1, "w": 2000},
               timeout=35
           )
           data = response.json().get("data", {})
           return data.get("markdown", "")
       except:
           return ""

   def synthesize(self, prompt):
       response = self.client.chat.completions.create(
           model="gpt-4",
           messages=[{"role": "user", "content": prompt}]
       )
       return response.choices[0].message.content

   def research(self, question):
       # Search
       results = self.search(question, num=8)

       # Extract
       contents = []
       for r in results[:5]:
           content = self.extract(r['url'])
           if content:
               contents.append(f"SOURCE: {r['title']}\nURL: {r['url']}\n\n{content[:2000]}")

       # Synthesize
       context = "\n\n---\n\n".join(contents)
       report = self.synthesize(f"""
Based on these sources, answer: {question}

{context}

Provide a comprehensive answer with citations.
""")

       return report

# Usage
if __name__ == "__main__":
   agent = MiniDeepResearch()

   question = input("What would you like to research? ")
   print("\nResearching...\n")

   report = agent.research(question)
   print(report)

   # Save
   with open("report.md", "w") as f:
       f.write(f"# Research Report\n\n**Question**: {question}\n\n{report}")

   print("\nReport saved to report.md")

Testing Your Agent

python research_agent.py

Example questions to try:

• “What are the top AI trends in 2025?”

• “Compare React vs Vue.js for enterprise applications”

• “Analyze the electric vehicle market in Europe”

Cost Estimation

For a typical research query:

SERP API: 10 searches x 1 credit = 10 credits
Reader API: 5 extractions x 2 credits = 10 credits
Total SearchCans: 20 credits (~$0.011 at $0.56/1,000 credits on the Ultimate plan)
GPT-4: ~100K tokens x $30/1M = $3.00

Total per research: ~$3.01 (dominated by LLM inference, not data acquisition)

Optimization tips:

• Use GPT-3.5 for simple syntheses ($0.50 vs $3)

• Cache search results

• Limit max sources for quick research

Next Steps

Enhancements to add:

1. Source credibility scoring

1. Multi-language support

1. PDF export

1. Web interface (Flask/Streamlit)

1. Scheduled research (cron jobs)

1. Collaborative features

Troubleshooting

Common issues:

# Issue: API key not found
# Solution: Check .env file and load_dotenv()

# Issue: Timeout errors
# Solution: Increase timeout, add retry logic

def extract_with_retry(self, url, retries=3):
   for attempt in range(retries):
       try:
           return self.extract(url)
       except requests.Timeout:
           if attempt == retries - 1:
               return ""
           time.sleep(2 ** attempt)  # Exponential backoff

You now have a functional DeepResearch agent! Experiment, enhance, and build amazing research tools.

Related Resources

DeepResearch Series:

• What is DeepResearch

• Architecture Deep Dive

• Beyond RAG

• Business Applications

API Documentation:

• SERP API Guide

• Reader API Guide

• API Docs

Get Started:

• Sign up for SearchCans

• API Documentation

• Pricing

Build your own DeepResearch agent with SearchCans APIs. Start free with 100 free credits.