Document APIs can return text that is useful for finding research evidence, but a Markdown response is not a verified research table. Preserve the original document, the extraction output and the location of each value. Check headers, units and missing cells against the source before creating a structured dataset.
Request a document and preserve its version
The SearchCans document contract uses POST /api/v1/url with t:"url", the reviewed document URL in s and file:1. Validate HTTP and business status, then read data.fileMarkdown. Keep the source URL, retrieval time and a hash of the input file when you can obtain it. Do not substitute data.markdown for the document field.
Use a table fixture with explicit units
import re
def document_markdown(envelope):
if envelope.get("code") != 0 or not isinstance(envelope.get("data"), dict):
raise ValueError("Document response failed")
text = envelope["data"].get("fileMarkdown")
if not isinstance(text, str) or not text.strip():
raise ValueError("Missing fileMarkdown")
return text
def checked_table(markdown, expected_header, source_id):
lines = [line.strip() for line in markdown.splitlines() if line.strip().startswith("|")]
split = lambda line: [cell.strip() for cell in line.strip("|").split("|")]
if len(lines) < 3 or split(lines[0]) != expected_header:
raise ValueError("Missing or different table header")
if not all(re.fullmatch(r":?-{3,}:?", cell) for cell in split(lines[1])):
raise ValueError("Missing Markdown table separator")
rows = [split(line) for line in lines[2:]]
if any(len(row) != len(expected_header) or any(not cell for cell in row) for row in rows):
raise ValueError("Missing cell: preserve raw output for review")
return {"source_id":source_id, "header":expected_header, "rows":rows,
"status":"requires_source_comparison"}
if __name__ == "__main__":
# Owned fictional document fragment. No scientific measurement or OCR result.
text = "# Sample protocol\n| Group | Dose (mg) |\n| --- | --- |\n| A | 5 |"
print(checked_table(text, ["Group", "Dose (mg)"], "owned-fixture:section-1"))
This owned fixture is fictional and does not report an OCR benchmark. It returns one row with a dose of five milligrams and the source ID owned-fixture:section-1. Removing the dose cell rejects the row. The parser handles a simple pipe table only; escaped pipes, merged cells, footnotes and multiline cells require another parser or source review.
Keep uncertainty attached to the value
For a real paper, record page and table numbers from the original, the extracted passage, the unit and the review decision. Do not infer a page number when the extraction lacks it. A superscript may identify a note rather than a value; a blank cell may mean not measured rather than zero. Both distinctions need the source. Keep null values and reasons until a reviewer resolves them.
| Check | Failing example | Next action |
|---|---|---|
| Header and unit | Dose returned without mg | Review original table heading |
| Row alignment | A group name paired with the next row’s value | Reject automated row association |
| Source location | No page/table identifier available | Retain URL and require manual locator |
| Permission and provenance | Unclear reuse permission | Review terms before further use |
Evaluate real extraction with a versioned, permitted document set and publish the raw failures before claiming accuracy. The local parser test only establishes that the checks reject their designed defects. For source-record design use training-data provenance records; for webpage extraction selection use the Reader guide.
When the output must be a printed document rather than extracted text, use webpage-to-PDF readiness and verification and inspect the PDF itself.