Tutorial 4 min read

Extract Research Tables: Document Fields, Units and Review

Extract document Markdown, preserve research-table units and source locations, and reject missing cells. Includes a clearly labelled fictional table fixture.

(Updated: ) • 604 words

Document APIs can return text that is useful for finding research evidence, but a Markdown response is not a verified research table. Preserve the original document, the extraction output and the location of each value. Check headers, units and missing cells against the source before creating a structured dataset.

Request a document and preserve its version

The SearchCans document contract uses POST /api/v1/url with t:"url", the reviewed document URL in s and file:1. Validate HTTP and business status, then read data.fileMarkdown. Keep the source URL, retrieval time and a hash of the input file when you can obtain it. Do not substitute data.markdown for the document field.

Use a table fixture with explicit units

import re

def document_markdown(envelope):
    if envelope.get("code") != 0 or not isinstance(envelope.get("data"), dict):
        raise ValueError("Document response failed")
    text = envelope["data"].get("fileMarkdown")
    if not isinstance(text, str) or not text.strip():
        raise ValueError("Missing fileMarkdown")
    return text

def checked_table(markdown, expected_header, source_id):
    lines = [line.strip() for line in markdown.splitlines() if line.strip().startswith("|")]
    split = lambda line: [cell.strip() for cell in line.strip("|").split("|")]
    if len(lines) < 3 or split(lines[0]) != expected_header:
        raise ValueError("Missing or different table header")
    if not all(re.fullmatch(r":?-{3,}:?", cell) for cell in split(lines[1])):
        raise ValueError("Missing Markdown table separator")
    rows = [split(line) for line in lines[2:]]
    if any(len(row) != len(expected_header) or any(not cell for cell in row) for row in rows):
        raise ValueError("Missing cell: preserve raw output for review")
    return {"source_id":source_id, "header":expected_header, "rows":rows,
            "status":"requires_source_comparison"}

if __name__ == "__main__":
    # Owned fictional document fragment. No scientific measurement or OCR result.
    text = "# Sample protocol\n| Group | Dose (mg) |\n| --- | --- |\n| A | 5 |"
    print(checked_table(text, ["Group", "Dose (mg)"], "owned-fixture:section-1"))

This owned fixture is fictional and does not report an OCR benchmark. It returns one row with a dose of five milligrams and the source ID owned-fixture:section-1. Removing the dose cell rejects the row. The parser handles a simple pipe table only; escaped pipes, merged cells, footnotes and multiline cells require another parser or source review.

Keep uncertainty attached to the value

For a real paper, record page and table numbers from the original, the extracted passage, the unit and the review decision. Do not infer a page number when the extraction lacks it. A superscript may identify a note rather than a value; a blank cell may mean not measured rather than zero. Both distinctions need the source. Keep null values and reasons until a reviewer resolves them.

Check Failing example Next action
Header and unit Dose returned without mg Review original table heading
Row alignment A group name paired with the next row’s value Reject automated row association
Source location No page/table identifier available Retain URL and require manual locator
Permission and provenance Unclear reuse permission Review terms before further use

Evaluate real extraction with a versioned, permitted document set and publish the raw failures before claiming accuracy. The local parser test only establishes that the checks reject their designed defects. For source-record design use training-data provenance records; for webpage extraction selection use the Reader guide.

When the output must be a printed document rather than extracted text, use webpage-to-PDF readiness and verification and inspect the PDF itself.

Sources and reproduction

Tags:

Tutorial Web Scraping AI Agent API Development Integration
SearchCans Team

SearchCans Team

SERP API & Reader API Experts

The SearchCans engineering team builds high-performance search APIs serving developers worldwide. We share practical tutorials, best practices, and insights on SERP data, web scraping, RAG pipelines, and AI integration.

Ready to build with SearchCans?

Test SERP API and Reader API with 100 free credits. No credit card required.