Refreshing a RAG source should replace obsolete evidence, not simply append another copy. Keep a source revision hash, the derived chunk identities and a removal state. A source that disappears needs a tombstone path, even when its last stored content is recent.
Detect a revision change deterministically
from hashlib import sha256
def reconcile_version(url, previous_text, current_text):
old, new = sha256(previous_text.encode()).hexdigest(), sha256(current_text.encode()).hexdigest()
return {'url': url, 'old_sha256': old, 'new_sha256': new,
'action': 'replace_and_invalidate_old_chunks' if old != new else 'keep'}
changed = reconcile_version("https://example.org/docs", "revision A", "revision B")
assert changed["action"] == "replace_and_invalidate_old_chunks"The local case detects text changes. It does not perform the index update or establish that a semantically equivalent page must be re-embedded. The storage adapter owns that policy.
Record the replacement transaction
| Record | Purpose |
|---|---|
| Source URL and new hash | Identify the accepted revision |
| Old hash and chunk IDs | Locate obsolete material |
| Replacement status | Keep a partially completed update visible |
| Removal/tombstone status | Prevent withdrawn evidence from resurfacing |
| Rollback metadata | Recover the prior accepted version when appropriate |
Test removal and failed refresh separately
A timeout does not establish that a source was removed. Keep a failed-refresh state and follow the application’s stale-data policy. A confirmed removal should locate affected chunks and caches through a controlled storage operation.
Build local cases for an unchanged revision, a changed revision, a failed update and a removed source. Then verify that old chunks cannot coexist as current evidence after a successful replacement. No universal hourly or daily refresh interval is assumed.
Reference documentation: www.searchcans.com — /reader-api; developers.llamaindex.ai — /python/framework/understanding/rag.
