Validate Markdown Before Ingesting It Into a RAG Index

Validate extracted Markdown before it enters a RAG index. A source can be readable while missing the table or code needed for its task. Keep accept, quarantine and failure states separate so a downstream answer does not inherit unexamined extraction gaps.

Use a source acceptance record

Test content loss deliberately

Create an owned source with a paragraph, table and code token. Remove one required item from the extracted candidate and confirm that it is quarantined. Empty Markdown should not create an accepted document, while a valid document with no answer to a particular query is a later retrieval issue.

Keep table relationships and code meaning where the task requires them. If the extraction output cannot preserve the required structure, store the source evidence or choose another permitted extraction route rather than reporting a clean success.

Update the source version atomically

from hashlib import sha256

def reconcile_version(url, previous_text, current_text):
    old, new = sha256(previous_text.encode()).hexdigest(), sha256(current_text.encode()).hexdigest()
    return {'url': url, 'old_sha256': old, 'new_sha256': new,
            'action': 'replace_and_invalidate_old_chunks' if old != new else 'keep'}

assert reconcile_version("https://example.org/docs", "old", "new")["action"] == "replace_and_invalidate_old_chunks"
assert reconcile_version("https://example.org/docs", "same", "same")["action"] == "keep"

The hash fixture identifies a changed revision. Actual index replacement must use the storage system’s supported operations and preserve rollback metadata. Hash equality does not establish that a source remains legally or contextually suitable.

Reference documentation: www.searchcans.com — /reader-api; developers.llamaindex.ai — /python/framework/understanding/rag.

Related implementation and validation guides

• LLM Web Crawlers: Define the Ingestion Contract First