A successful web-extraction response can still be incomplete for your task. Check the passages, table cells, units, code and source links that the downstream answer needs. Treat missing information as a reviewable failure, even when the HTTP request and API business code both indicate success.
Use three different kinds of sample
| Sample | Required checks | Failure to record |
|---|---|---|
| Static article | Main heading, requested passage and citations | Navigation text mistaken for main content |
| JavaScript-dependent page | Visible task-specific text after the selected rendering mode | Content absent from a static response |
| Table or code page | Header, unit, exact cells and code indentation | Readable prose with missing evidence |
For an actual extraction test, save the reviewed source version, URL and time; document whether browser rendering was used. Keep the original and extracted output side by side. The fixture below tests the acceptance logic on owned synthetic Markdown. It is not a measurement of SearchCans extraction accuracy or a claim that JavaScript content was fetched.
Reject missing task evidence explicitly
def check_extraction(markdown, expected):
if not isinstance(markdown, str) or not markdown.strip():
return {"status":"reject", "missing":["nonempty_body"]}
missing = [name for name, literal in expected.items() if literal not in markdown]
return {"status":"review" if missing else "fixture_pass", "missing":missing}
if __name__ == "__main__":
# Owned synthetic Markdown; this tests the acceptance logic, not an extraction vendor.
text = "# Trial\nDose: 5 mg\n| Group | n |\n| A | 8 |\n```python\nprint(1)\n```\n[Protocol](https://example.com/protocol)"
expected = {"main_heading":"# Trial", "unit":"5 mg", "table_row":"| A | 8 |",
"code":"print(1)", "source_link":"https://example.com/protocol"}
print(check_extraction(text, expected))
print(check_extraction(text.replace("| A | 8 |", ""), expected))
The intact fixture returns fixture_pass. Removing its table row returns review with table_row missing. That distinction lets a pipeline retain the source and failure log without letting an LLM fill in the absent value. Do not convert a failed check into an empty string that looks like a real result.
Separate content quality from service and policy checks
For SearchCans, check the HTTP response, business code and data.markdown separately. Its Reader documentation defines rendering and proxy options; proxy:0 means no proxy. A selected mode is not evidence that every page section survived. Record actual usage according to the current billing contract rather than estimating savings from output length.
Extraction alone does not establish permission to reuse text, a provider’s retention policy, or compliance with a privacy law. Evaluate those matters against the applicable source terms, your own data handling and the provider’s current policy. This article does not assert that no data is stored or cached.
For format checks after extraction, use Markdown ingestion acceptance tests. For choosing an extraction service, use the URL-to-Markdown API selection guide. This page owns completeness verification, not vendor ranking.
