# RAG chunking strategies: document structure versus fixed token windows

A RAG chunking strategy defines which evidence can be retrieved together, not just how many tokens enter an embedding call.

Author: Retrivium team
Updated: 2026-11-03
Canonical URL: https://retrivium.ai/journal/rag-chunking-structure-token-boundaries/
Language: en

A RAG chunking strategy defines which evidence can be retrieved together, not just how many tokens enter an embedding call. Compare structural boundaries and fixed windows against the same questions before choosing a default size or overlap.

## Start from the document relationship

A fixed window can divide a definition from the exception immediately following it. Structural chunking can preserve a section but still produce an oversized unit. The [Docling chunking concepts](https://docling-project.github.io/docling/concepts/chunking/) distinguish hierarchical units from tokenizer-aware refinement. Neither design automatically establishes semantic completeness. Preserve the original document structure so a failed retrieval can be traced back to the evidence boundary that was actually encoded.

## Count the actual embedding input

The [hybrid-chunking example](https://docling-project.github.io/docling/_generated/examples/hybrid_chunking/) emphasizes using the embedding model's tokenizer. A raw text length is not a reliable token budget, especially when headings or other metadata are added. Inspect the final serialized input delivered to the encoder. A chunk that fits before contextualization may exceed the intended limit afterward. Silent truncation can then remove evidence while the pipeline still reports a successfully generated vector.

## Keep hierarchy and source identity

The [Docling document representation](https://docling-project.github.io/docling/concepts/docling_document/) preserves sections, groups and provenance. Use those relationships when assigning source IDs and headings to chunks. Do not flatten every heading into unrelated repeated text or treat page furniture as the section body. A result should identify both its retrievable unit and its source document, making it possible to reconstruct the passage used as evidence rather than merely display an isolated string.

## Build a boundary-sensitive fixture

Use an illustrative manual containing a short definition, a later qualification, a table and a section transition. Write questions that require each relationship. Generate structural and fixed-window variants from that same source, preserving the embedding model and retrieval settings. Judge whether the retrieved text contains enough evidence, not whether a chunk happens to include the query words.

| Strategy property | Failure to inspect |
|---|---|
| Fixed window | A supporting relationship is cut in half |
| Structural unit | The unit exceeds the effective encoder budget |
| Metadata enrichment | Added context changes length or meaning |

## Compare equivalent retrieval budgets

Changing chunk size changes the number of indexed units and the amount of text returned by a fixed hit limit. Compare source coverage and the final evidence budget explicitly rather than keeping only the same number of hits. A configuration returning longer passages may appear better because it supplies more source text. Preserve duplicates and missing context as separate observations so the experiment can explain the tradeoff.

## Choose rules with visible exceptions

Document how headings, tables, very long sections and short fragments are handled. Retain the source revision and a small set of boundary cases with expected evidence. A practical chunking policy can use different treatment for different structural units. Its value comes from preserving usable evidence within a known input budget, not from claiming that one token count is universally optimal for every document or embedding family.

Related reading: [A local RAG server: placing components around the GPU](/journal/rag-server-local-architecture/); [Batch text embeddings on your own GPU](/journal/text-embeddings-batch/).