When ingesting web Markdown into LlamaIndex, attach source identity before building the index and verify that it remains on retrieved nodes. A chunk without its URL and revision context is harder to audit, refresh or remove. Begin with a small deterministic ingestion fixture.
Check metadata with installed LlamaIndex
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.embeddings import MockEmbedding
document = Document(
text="The fixture supports version 2.",
metadata={"url": "https://example.org/docs", "checked_at": "2026-10-10"},
)
index = VectorStoreIndex.from_documents([document], embed_model=MockEmbedding(embed_dim=8))
nodes = index.as_retriever(similarity_top_k=1).retrieve("Which version?")
assert nodes[0].node.metadata["url"] == "https://example.org/docs"This exact contract was tested with llama-index-core 0.14.25. MockEmbedding avoids a model service and verifies ingestion/metadata plumbing. It is not a semantic-relevance or answer-quality test.
Carry the revision through refresh
A fixed refresh interval is not suitable for every source. Use the task’s freshness requirement and record when the evidence was checked. Keep a source-reading failure separate from a missing document in the index.
- Keep URL, access time, content hash and extraction status before constructing Document.
- Validate required source content before accepting the Markdown.
- On a changed revision, invalidate the old nodes through the index’s supported storage operations.
- On removal, mark the source and locate the affected nodes instead of only adding a new copy.
Add quality evaluation after contract validation
Replace mock embeddings with the configured production embedding model only after the source contract passes. Evaluate frozen questions and labelled supporting spans, keeping model versions and candidate budgets fixed. Successful indexing does not show that later answers are correct.
Reference documentation: developers.llamaindex.ai — /python/framework/understanding/rag; www.searchcans.com — /reader-api.
