A web-evidence token budget is a selection problem, not just a format conversion. Keep the passages needed to answer the question, retain their source identities and expose what was omitted. Measure the serialized evidence with the chosen tokenizer. If required evidence cannot fit, change the question or budget instead of silently dropping it.
Assign required evidence before trimming text
| Record | Reason |
|---|---|
| Source URL and passage ID | A selected statement remains traceable |
| Required flag | Evidence necessary for the task cannot disappear silently |
| Priority | Optional context has an explicit selection order |
| Reserved context | Instructions, history, tools and output allowance need their own accounting |
| Omission reason | The consumer can distinguish missing evidence from a negative finding |
The example retains complete source-linked records rather than clip arbitrary characters. It re-tokenizes the complete serialized candidate because concatenation boundaries can change token counts. Its cl100k_base choice is a local demonstration tokenizer, not a claim about every model’s billing or context window.
Test selection against the exact serialization
Install tiktoken and run the standalone Python example. The required and optional passages are synthetic; the output shows which record was omitted. Supply the actual tokenizer and measured reserved context when adapting it to a model request.
import json
def serialized(records):
return json.dumps(records, ensure_ascii=False, separators=(",", ":"))
def select_evidence(records, encode, context_limit, reserved_tokens):
if context_limit <= reserved_tokens or reserved_tokens < 0:
raise ValueError("No evidence budget")
budget = context_limit - reserved_tokens
if len(encode(serialized([]))) > budget:
raise ValueError("Empty evidence serialization does not fit")
ordered = sorted(records, key=lambda r: (not r.get("required", False),
-r.get("priority", 0), r["id"]))
kept, omitted = [], []
for record in ordered:
candidate = {"id": record["id"], "url": record["url"], "text": record["text"]}
proposed = kept + [candidate]
if len(encode(serialized(proposed))) <= budget:
kept = proposed
elif record.get("required", False):
raise ValueError("Required evidence does not fit; change question or budget")
else:
omitted.append({"id": record["id"], "reason": "evidence_budget"})
return {"evidence": kept, "omitted": omitted,
"evidence_tokens": len(encode(serialized(kept))),
"evidence_budget": budget,
"billing_estimate": "not_computed"}
if __name__ == "__main__":
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
fixtures = [
{"id": "required", "url": "https://example.test/a", "text": "Synthetic required passage.",
"required": True, "priority": 10},
{"id": "optional", "url": "https://example.test/b", "text": "Extra synthetic context. " * 100,
"required": False, "priority": 1}]
print(json.dumps({"tokenizer": "cl100k_base", "fixture_only": True,
"result": select_evidence(fixtures, enc.encode, 200, 100)}, indent=2))
Reject an incomplete answer budget
If a required record does not fit, the function raises an error. It does not return a smaller evidence set and call it complete. The caller can narrow the question, split a multi-part request, increase available context where supported or ask for a different output. That decision should remain visible in the job record.
The function counts only its own serialized evidence. It does not independently measure the system prompt, conversation, tool schema, attachments or output reserve. Those are caller responsibilities. Verify the final provider request as well; a locally fitting evidence string is not proof that the complete request will fit.
Check source fidelity independently of size
A short passage can omit a date, exception or unit that changes the answer. Keep those qualifiers with the selected claim and test that its source URL still resolves to the reference evidence. Compare HTML and Markdown token measurement when the question is format overhead, and context-window engineering for broader context organization.
Use URL-to-Markdown structure checks before selection. This page owns query-specific evidence budgeting, not a universal token-reduction percentage or a promise that Markdown improves answer accuracy.
