---
topic: ai-engineering
author: Crashtech Editorial
published: Oct 17, 2026 · read: 2 min
---

Choosing a Chunking Strategy for RAG: Fixed, Semantic, Late, and Hierarchical

A comprehensive architectural evaluation of eight document splitting strategies for RAG agents: trade-offs across retrieval recall, cost, and boundary noise.

– –

In Retrieval-Augmented Generation (RAG), engineers spend weeks tweaking embedding models, vector database index algorithms (HNSW vs IVF-PQ), and system prompts. Yet, the single highest-leverage factor dictating whether a retrieval system succeeds or hallucinates is the unglamorous choice of Document Chunking.

The chunk is the fundamental atomic currency of retrieval. If a chunk is too small, it strips away the surrounding context, turning a clear causal statement into an ambiguous fragment. If a chunk is too large, it dilutes the semantic embedding vector with irrelevant noise, dragging down similarity scores and introducing the Lost-in-the-Middle blindspot.

There is no universal “one-size-fits-all” chunking strategy. An architecture processing technical API references requires completely different document splitting than one ingesting legal contracts, financial earnings transcripts, or narrative customer support tickets.

The Eight Chunking Strategies

Production systems select from eight primary document splitting paradigms, balancing computational complexity against semantic retrieval accuracy:

+-----------------------------------------------------------------------------+
|                        THE EIGHT CHUNKING STRATEGIES                        |
+-----------------------------------------------------------------------------+

 1. Fixed-Size Chunking with Overlap (Baseline)
    - Splits strictly every N characters/tokens with M tokens of overlap.
    - Cost: Free | Quality: Poor (Cuts sentences and words in half).

 2. Recursive Character Splitting (Standard Default)
    - Attempts to split on paragraphs ('\n\n'), then sentences ('\n'), then words.
    - Cost: Negligible | Quality: Good baseline for generic text.

 3. Document-Structure-Aware Chunking (Markdown / HTML / Code)
    - Respects document AST headers (# H1, ## H2), tables, and function blocks.
    - Cost: Low | Quality: High for technical documentation and code.

 4. Semantic Boundary Chunking (Embedding Distance Shifts)
    - Computes sentence embeddings and splits when cosine similarity between 
      adjacent sentences drops below a statistical threshold.
    - Cost: Moderate (Local encoder required) | Quality: High semantic cohesion.

 5. Propositional Chunking (LLM-Decomposed Statements)
    - An LLM rewrites complex text into standalone, self-contained factual facts.
    - Cost: High | Quality: Exceptional for entity-dense question answering.

 6. Agentic Adaptive Chunking
    - An agent inspects the document type and dynamically selects splitting rules.
    - Cost: High | Quality: Essential for multi-modal, highly varied corpora.

 7. Late Chunking (Full-Context Post-Embedding Splitting)
    - Embeds the ENTIRE document through a long-context transformer first, 
      then pools token embeddings into chunks preserving global document attention.
    - Cost: High (Long-context forward pass) | Quality: State-of-the-art context retention.

 8. Hierarchical (Parent-Child) Chunking
    - Small child chunks (100 tokens) are indexed for precision vector search; 
      retrieval expands to return the full parent section (1,000 tokens) to the LLM.
    - Cost: Low | Quality: Best operational tradeoff for enterprise RAG.

Deep Dive: Late Chunking vs Traditional Chunking

The most significant recent breakthrough in chunking architecture is Late Chunking (introduced by Jina AI):

[Traditional Chunking]
  Document ---> Split into Chunks ---> Embed Each Chunk Separately
  Flaw: When Chunk 2 is embedded, the encoder cannot see Chunk 1. 
        Pronouns ("It", "They", "The system") lose all referential context!

[Late Chunking (State of the Art)]
  Document ---> Pass Entire Document through Long-Context Transformer
           ---> Generate Token Embeddings with Full Document Attention
           ---> Mean-Pool Token Vectors into Chunk Boundaries
  Advantage: Every chunk vector contains rich contextual signals from the 
             entire surrounding document!

In traditional chunking, if Chunk 1 states: “Apple introduced the M3 chip in 2023”, and Chunk 2 states: “It features an 8-core CPU and 10-core GPU”, Chunk 2’s vector embedding has no idea what “It” refers to. In Late Chunking, the bidirectional cross-attention mechanism resolves the pronoun “It” to “Apple M3” before chunk boundaries are partitioned.

Deep Dive: Hierarchical Parent-Child Chunking

For 85% of enterprise production applications, Hierarchical Chunking offers the best balance of retrieval recall and generative quality without the high compute cost of late chunking:

+-----------------------------------------------------------------------------+
|                     HIERARCHICAL PARENT-CHILD CHUNKING                      |
+-----------------------------------------------------------------------------+

 [Parent Chunk: Entire Policy Section (~1,200 tokens)]
  "Section 4.2: Enterprise Travel Reimbursement Guidelines..."
        |
        +---> Child Chunk A (150 tokens): "Meal per-diem allowances: $75/day"
        +---> Child Chunk B (150 tokens): "Flight booking rules & class limits"
        +---> Child Chunk C (150 tokens): "Hotel cancellation & receipt rules"

 Vector Search: Compares user query ONLY against small Child Chunks.
 Prompt Payload: When Child A matches, the system retrieves and injects 
                 the ENTIRE Parent Chunk into the LLM context!

Why Hierarchical Splitting Dominates

  • High Vector Distinctiveness: Small child chunks produce tight, high-density vector embeddings that match specific keywords and entities accurately.
  • Complete Contextual Synthesis: When the generative LLM receives the prompt, it gets the entire parent section, eliminating fragmented sentences and missing prerequisites.

Comparative Decision Matrix

StrategyCompute CostBest Suited CorpusPrimary Failure Mode
Fixed / OverlapZeroPre-indexing massive logsSentence amputation, semantic loss
RecursiveVery LowGeneral blogs, clean articlesDisregards tabular structures
Document StructureLowGitHub repos, API docs, MarkdownMisses unstructured PDF text
Semantic BoundaryMediumTranscripts, unstructured dialoguesErratic, variable chunk lengths
PropositionalHigh (LLM calls)Knowledge graphs, dense fact sheetsExpensive ingestion pipelines
Late ChunkingHigh (GPU pass)Highly contextual long documentsRequires 8k+ context embedding model
HierarchicalLowCorporate HR, legal contracts, SLAs2x storage overhead in vector DB

Architectural Takeaway

Stop splitting documents using arbitrary 500-token fixed windows. When building production RAG systems:

  • Parse document structures (Markdown, HTML, ASTs) natively to keep code functions and tables intact.
  • Implement Hierarchical Parent-Child chunking as your enterprise default: index small granular child chunks for search accuracy, and deliver full parent contexts to the model.
  • Adopt Late Chunking for highly contextual corpora where pronoun resolution and document-wide narrative continuity dominate retrieval success.
Advertisement

/* Comments */