RAG Markdown Ingestion: Acceptance Tests and Failure Logs

Markdown ingestion needs an acceptance contract: the source and retrieval time must survive, and the headings, tables and code required by the task must remain usable. This page provides a small deterministic validator. It does not select a vendor or convert a URL; those are separate stages.

Specify requirements from the source

For each sample, name the literals that must remain: a table row with its unit, a code statement, or a cited source URL. Make these requirements before inspecting the output. Otherwise a validator can be made to pass simply by checking whichever content happened to survive. Heading-level and fence checks catch some structural defects, but they do not prove factual completeness.

Run an acceptance check

import re

def accept_markdown(record, requirements):
    text = record.get("markdown")
    if not isinstance(text, str) or not text.strip():
        return {"accepted":False, "failures":["empty_markdown"]}
    failures = []
    if not record.get("source_url") or not record.get("retrieved_at"):
        failures.append("missing_provenance")
    headings = re.findall(r"^(#{1,6})\s+(.+)$", text, re.M)
    levels = [len(level) for level, _ in headings]
    if any(b > a + 1 for a, b in zip(levels, levels[1:])):
        failures.append("heading_level_skip")
    if sum(1 for line in text.splitlines() if line.lstrip().startswith("```")) % 2:
        failures.append("unclosed_code_fence")
    for name, literal in requirements.items():
        if literal not in text:
            failures.append("missing_" + name)
    return {"accepted":not failures, "failures":failures}

if __name__ == "__main__":
    fixture = {"source_url":"https://example.com/guide", "retrieved_at":"2026-10-10T00:00:00Z",
               "markdown":"# Dose\n## Table\n| Dose | Unit |\n| 5 | mg |\n```python\nprint(5)\n```"}
    print(accept_markdown(fixture, {"dose_row":"| 5 | mg |"}))

The synthetic fixture passes. Remove its retrieval time and it fails provenance; remove the final code fence and it fails structure. A missing required row fails independently. These tests exercise the validator, not the quality of any provider’s extraction. Run the same contract on saved outputs from your reviewed source set to evaluate real ingestion.

Record a failure without inventing a replacement

Save source_url, retrieved_at, the output hash, validator version and failure names. Quarantine failed records before chunking. If a table cell is missing, retain a null and its failure reason; do not ask an LLM to manufacture a plausible number. A later corrected extraction should create a new version and invalidate chunks from the failed version.

This literal-based check will miss reordered information, paraphrases and incorrectly associated cells. Add task-specific parsing or human review for those cases. Keep a versioned list of expected failures so that “more accepted outputs” does not hide weaker checks. For extraction completeness use the missing-content method; for conversion use the Python URL conversion tutorial; for service selection use the Reader selection guide.

Sources and reproduction

• Reader output contract