Evaluate a RAG chunking strategy against fixed questions and labelled source spans. A chunk size that preserves one answer may split another table or detach a claim from its qualification. Choose from observed evidence retention and retrieval behavior, not a universal token range.
Freeze the evidence before comparing settings
| Evaluation input | Why it stays fixed |
|---|---|
| Document revision and hash | Prevent source changes from explaining the result |
| Questions and required source spans | Define what a successful retrieval must preserve |
| Chunking candidates | Identify the setting actually changed |
| Embedding/retrieval configuration | Avoid changing two variables at once |
| Candidate count and context budget | Keep the retrieval opportunity comparable |
Score the failures that chunking can cause
For each question, record whether the returned chunks contain the required answer span and any qualification needed to interpret it. Check tables, code blocks, lists and statements that depend on the previous paragraph. An answer-looking sentence without its constraint can be a support failure.
Compare overlap and chunk boundaries using the same labelled corpus. Keep a count of duplicate content and the resulting context size alongside evidence retention. Do not claim that heading splitting or a particular token range always outperforms another strategy.
A reproducible acceptance exercise
This page proposes an evaluation method. It does not report an unperformed model benchmark or assume that smaller chunks reduce every application’s cost. The ingestion tutorial covers moving source text into the index.
- Create an owned document containing a claim, its limitation, a table and a code example.
- Label questions that require each of those elements.
- Run candidate chunking settings and retain the actual chunks.
- Reject settings that detach required qualifications even if they produce fewer tokens.
Reference documentation: developers.llamaindex.ai — /python/framework/understanding/rag.
