Choosing a Chunking Strategy for RAG: Fixed, Semantic, Late, and Hierarchical
A comprehensive architectural evaluation of eight document splitting strategies for RAG agents: trade-offs across retrieval recall, cost, and boundary noise.
On this page
In Retrieval-Augmented Generation (RAG), engineers spend weeks tweaking embedding models, vector database index algorithms (HNSW vs IVF-PQ), and system prompts. Yet, the single highest-leverage factor dictating whether a retrieval system succeeds or hallucinates is the unglamorous choice of Document Chunking.
The chunk is the fundamental atomic currency of retrieval. If a chunk is too small, it strips away the surrounding context, turning a clear causal statement into an ambiguous fragment. If a chunk is too large, it dilutes the semantic embedding vector with irrelevant noise, dragging down similarity scores and introducing the Lost-in-the-Middle blindspot.
There is no universal “one-size-fits-all” chunking strategy. An architecture processing technical API references requires completely different document splitting than one ingesting legal contracts, financial earnings transcripts, or narrative customer support tickets.
The Eight Chunking Strategies
Production systems select from eight primary document splitting paradigms, balancing computational complexity against semantic retrieval accuracy:
+-----------------------------------------------------------------------------+
| THE EIGHT CHUNKING STRATEGIES |
+-----------------------------------------------------------------------------+
1. Fixed-Size Chunking with Overlap (Baseline)
- Splits strictly every N characters/tokens with M tokens of overlap.
- Cost: Free | Quality: Poor (Cuts sentences and words in half).
2. Recursive Character Splitting (Standard Default)
- Attempts to split on paragraphs ('\n\n'), then sentences ('\n'), then words.
- Cost: Negligible | Quality: Good baseline for generic text.
3. Document-Structure-Aware Chunking (Markdown / HTML / Code)
- Respects document AST headers (# H1, ## H2), tables, and function blocks.
- Cost: Low | Quality: High for technical documentation and code.
4. Semantic Boundary Chunking (Embedding Distance Shifts)
- Computes sentence embeddings and splits when cosine similarity between
adjacent sentences drops below a statistical threshold.
- Cost: Moderate (Local encoder required) | Quality: High semantic cohesion.
5. Propositional Chunking (LLM-Decomposed Statements)
- An LLM rewrites complex text into standalone, self-contained factual facts.
- Cost: High | Quality: Exceptional for entity-dense question answering.
6. Agentic Adaptive Chunking
- An agent inspects the document type and dynamically selects splitting rules.
- Cost: High | Quality: Essential for multi-modal, highly varied corpora.
7. Late Chunking (Full-Context Post-Embedding Splitting)
- Embeds the ENTIRE document through a long-context transformer first,
then pools token embeddings into chunks preserving global document attention.
- Cost: High (Long-context forward pass) | Quality: State-of-the-art context retention.
8. Hierarchical (Parent-Child) Chunking
- Small child chunks (100 tokens) are indexed for precision vector search;
retrieval expands to return the full parent section (1,000 tokens) to the LLM.
- Cost: Low | Quality: Best operational tradeoff for enterprise RAG.
Deep Dive: Late Chunking vs Traditional Chunking
The most significant recent breakthrough in chunking architecture is Late Chunking (introduced by Jina AI):
[Traditional Chunking]
Document ---> Split into Chunks ---> Embed Each Chunk Separately
Flaw: When Chunk 2 is embedded, the encoder cannot see Chunk 1.
Pronouns ("It", "They", "The system") lose all referential context!
[Late Chunking (State of the Art)]
Document ---> Pass Entire Document through Long-Context Transformer
---> Generate Token Embeddings with Full Document Attention
---> Mean-Pool Token Vectors into Chunk Boundaries
Advantage: Every chunk vector contains rich contextual signals from the
entire surrounding document!
In traditional chunking, if Chunk 1 states: “Apple introduced the M3 chip in 2023”, and Chunk 2 states: “It features an 8-core CPU and 10-core GPU”, Chunk 2’s vector embedding has no idea what “It” refers to. In Late Chunking, the bidirectional cross-attention mechanism resolves the pronoun “It” to “Apple M3” before chunk boundaries are partitioned.
Deep Dive: Hierarchical Parent-Child Chunking
For 85% of enterprise production applications, Hierarchical Chunking offers the best balance of retrieval recall and generative quality without the high compute cost of late chunking:
+-----------------------------------------------------------------------------+
| HIERARCHICAL PARENT-CHILD CHUNKING |
+-----------------------------------------------------------------------------+
[Parent Chunk: Entire Policy Section (~1,200 tokens)]
"Section 4.2: Enterprise Travel Reimbursement Guidelines..."
|
+---> Child Chunk A (150 tokens): "Meal per-diem allowances: $75/day"
+---> Child Chunk B (150 tokens): "Flight booking rules & class limits"
+---> Child Chunk C (150 tokens): "Hotel cancellation & receipt rules"
Vector Search: Compares user query ONLY against small Child Chunks.
Prompt Payload: When Child A matches, the system retrieves and injects
the ENTIRE Parent Chunk into the LLM context!
Why Hierarchical Splitting Dominates
- High Vector Distinctiveness: Small child chunks produce tight, high-density vector embeddings that match specific keywords and entities accurately.
- Complete Contextual Synthesis: When the generative LLM receives the prompt, it gets the entire parent section, eliminating fragmented sentences and missing prerequisites.
Comparative Decision Matrix
| Strategy | Compute Cost | Best Suited Corpus | Primary Failure Mode |
|---|---|---|---|
| Fixed / Overlap | Zero | Pre-indexing massive logs | Sentence amputation, semantic loss |
| Recursive | Very Low | General blogs, clean articles | Disregards tabular structures |
| Document Structure | Low | GitHub repos, API docs, Markdown | Misses unstructured PDF text |
| Semantic Boundary | Medium | Transcripts, unstructured dialogues | Erratic, variable chunk lengths |
| Propositional | High (LLM calls) | Knowledge graphs, dense fact sheets | Expensive ingestion pipelines |
| Late Chunking | High (GPU pass) | Highly contextual long documents | Requires 8k+ context embedding model |
| Hierarchical | Low | Corporate HR, legal contracts, SLAs | 2x storage overhead in vector DB |
Architectural Takeaway
Stop splitting documents using arbitrary 500-token fixed windows. When building production RAG systems:
- Parse document structures (Markdown, HTML, ASTs) natively to keep code functions and tables intact.
- Implement Hierarchical Parent-Child chunking as your enterprise default: index small granular child chunks for search accuracy, and deliver full parent contexts to the model.
- Adopt Late Chunking for highly contextual corpora where pronoun resolution and document-wide narrative continuity dominate retrieval success.
/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.