RAG Chunking Strategies: Recursive vs Semantic vs Late Chunking, and How to Pick on Your Own Corpus

Most RAG teams spend weeks comparing embedding models and vector databases, then ship with chunk_size=1000, overlap=200 copied from a tutorial. That default is often where retrieval quality is quietly lost.

Published studies keep finding the same thing: chunking configuration can influence retrieval quality as much as, or more than, the embedding model you choose (Vectara's NAACL 2025 work is a commonly cited example). It is also the cheapest lever to change, because you control it entirely.

This post covers what the main chunking strategies do, where each one breaks, and a small harness to pick one using your own data instead of someone's blog benchmark.

The problem: a chunk is the unit of retrieval and the unit of meaning

Every chunk has to do two jobs at once:

Be findable. Its embedding must land near the queries that should retrieve it.
Be answerable. Once retrieved, it must contain enough context for the LLM to answer without guessing.

These pull in opposite directions. Small chunks embed precisely but lose context ("it increased by 12%" - what increased?). Large chunks keep context but dilute the embedding with unrelated content, so retrieval gets fuzzy and you burn context window on noise.

Every strategy below is a different way of trading off those two jobs.

The strategies

1. Fixed-size chunking

Split every N tokens, optionally with overlap. Fast, deterministic, and blind to structure: it will cut a sentence, a table, or a code block in half.

Use it as a floor, not a destination.

2. Recursive chunking

Try to split on the largest natural separator first (sections, then paragraphs, then sentences, then words) until chunks fit the target size. It respects structure when structure exists and degrades gracefully when it does not.

This is the sensible default. Most independent comparisons that include it find it hard to beat by a meaningful margin on general text, usually with chunks around 256-512 tokens and 10-20% overlap. Treat those numbers as a starting point to test, not a law.

3. Structure-aware chunking

If your documents have real structure (Markdown headings, HTML, PDF layouts parsed into sections, API docs, contracts with numbered clauses), split on that structure and keep the heading path with each chunk. A chunk that begins with Refund Policy > Enterprise Plans > Cancellation retrieves far better than the bare paragraph.

This usually beats clever embedding-based splitting on structured corpora, and it is cheap. Pair it with a good parser (see our guide on document parsing for RAG).

4. Semantic chunking

Embed each sentence, then start a new chunk wherever similarity between adjacent sentences drops below a threshold. In theory, chunks follow topic boundaries.

In practice, results are mixed. Some evaluations report gains in retrieval recall; others find it produces very small fragments that hurt end-to-end answer accuracy, and that it did not beat simple recursive splitting under their setup. It also costs an embedding call per sentence at ingestion time. If you use it, enforce a minimum chunk size or you will get a pile of one-sentence chunks.

5. Late chunking

Standard ("early") chunking splits first, then embeds each chunk in isolation. Late chunking runs a long-context embedding model over the whole document, then pools the token embeddings per chunk span. Each chunk's vector is computed with awareness of the surrounding document, so "it" and "the company" resolve properly.

Requirements: a long-context embedding model that exposes token-level outputs (Jina's embedding models popularized the technique), and documents that fit in its context window. It addresses the lost-context problem without an LLM call per chunk.

6. Contextual retrieval

Instead of changing how you split, use an LLM to prepend a one-or-two-sentence situating summary to each chunk before embedding. This costs an LLM call per chunk at ingestion, which prompt caching makes tolerable. We cover it in depth in our contextual retrieval post.

Comparison

Strategy	Ingestion cost	Respects structure	Fixes lost context	Main failure mode
Fixed-size	Very low	No	No	Cuts mid-sentence, mid-table
Recursive	Low	Partly	No	Orphaned pronouns, headless chunks
Structure-aware	Low	Yes	Partly (via heading path)	Needs clean parsing; fails on flat text
Semantic	Medium (embed per sentence)	By topic, not layout	No	Tiny fragments, threshold tuning
Late chunking	Medium (long-context pass)	Optional	Yes	Model/doc length limits
Contextual retrieval	High (LLM per chunk)	Optional	Yes	Cost, summary hallucination

Code: a chunking benchmark you can run tomorrow

The only benchmark that matters is the one on your corpus with your queries. You need a small golden set: 30-100 questions, each mapped to the document (or passage) that answers it. Our guide on retrieval evaluation with golden sets covers building one.

The harness below chunks a corpus with several strategies, retrieves top-k for each question, and reports recall@k. It uses langchain-text-splitters and sentence-transformers; check the official docs for current package names and APIs, as they change.

# pip install langchain-text-splitters sentence-transformers numpy
import numpy as np
from langchain_text_splitters import (
    RecursiveCharacterTextSplitter,
    CharacterTextSplitter,
)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-small-en-v1.5")

# corpus: {doc_id: full_text}
# golden: [{"question": str, "doc_id": str, "answer_span": str}]
# A retrieved chunk is a hit if it comes from the right doc AND contains
# the answer span. Doc-level matching alone hides chunking problems.

def chunkers(chunk_size_chars: int, overlap_chars: int):
    return {
        "fixed": CharacterTextSplitter(
            separator="", chunk_size=chunk_size_chars,
            chunk_overlap=overlap_chars),
        "recursive": RecursiveCharacterTextSplitter(
            chunk_size=chunk_size_chars, chunk_overlap=overlap_chars),
        "recursive_headings": RecursiveCharacterTextSplitter(
            separators=["\n## ", "\n### ", "\n\n", "\n", ". ", " "],
            chunk_size=chunk_size_chars, chunk_overlap=overlap_chars),
    }

def build_index(splitter, corpus):
    chunks, meta = [], []
    for doc_id, text in corpus.items():
        for piece in splitter.split_text(text):
            chunks.append(piece)
            meta.append(doc_id)
    emb = model.encode(chunks, normalize_embeddings=True, batch_size=64)
    return chunks, meta, emb

def recall_at_k(chunks, meta, emb, golden, k=5):
    hits = 0
    for g in golden:
        q = model.encode([g["question"]], normalize_embeddings=True)[0]
        top = np.argsort(-(emb @ q))[:k]
        if any(meta[i] == g["doc_id"] and g["answer_span"] in chunks[i]
               for i in top):
            hits += 1
    return hits / len(golden)

def run(corpus, golden, sizes=(800, 1500, 2400), overlap_ratio=0.15):
    rows = []
    for size in sizes:  # ~200 / 375 / 600 tokens for English text
        overlap = int(size * overlap_ratio)
        for name, splitter in chunkers(size, overlap).items():
            chunks, meta, emb = build_index(splitter, corpus)
            rows.append({
                "strategy": name,
                "chunk_chars": size,
                "n_chunks": len(chunks),
                "recall@5": round(recall_at_k(chunks, meta, emb, golden), 3),
            })
    return rows

Two details that matter:

Match on the answer span, not just the document. If you only check doc ID, a chunk from the right document that does not contain the answer counts as a hit, and every strategy looks equally good.
Report n_chunks alongside recall. A strategy that gains two points of recall while tripling your index size and ingestion bill may not be a win.

Add your semantic, late-chunking, or contextual variants to chunkers() as extra strategies, using the same harness, so the comparison stays apples to apples.

Checklist before you ship a chunking config

Golden set of real user questions (not LLM-generated from the chunks themselves, which biases toward your current chunking)
Recall@k measured separately from answer accuracy, so you know whether retrieval or generation is failing
Chunk size swept across at least three values
Heading path or document title attached to every chunk
Tables, code blocks, and lists kept intact rather than split
Chunk metadata (doc ID, section, offset) stored so you can expand to neighbors or the parent section at query time
Re-run when you change embedding model, since the best chunk size shifts with it

Common pitfalls

Tuning on benchmarks you did not run. Published comparisons use different corpora, metrics, and judges, and many come from vendors selling a particular method. Several recent academic evaluations find limited evidence that elaborate chunking beats a well-configured recursive splitter across diverse corpora. Use them to generate hypotheses, then test.

Ignoring chunk size in tokens vs characters. Splitters count characters by default; embedding models have token limits. Check that your chunks are not being silently truncated by the embedder.

Overlap as a crutch. High overlap duplicates content, inflates the index, and fills top-k with near-identical chunks. A better fix for boundary problems is retrieving neighbor chunks or the parent section after the match (small-to-big retrieval).

Chunking mixed content uniformly. A policy PDF, a CSV export, and a Slack thread should not share one splitter. Route by document type.

Over-engineering before measuring. If recursive plus heading paths gets your recall within a point or two of semantic chunking, take the cheaper pipeline. Spend the saved effort on reranking or hybrid search, which usually move the needle further.

Next steps

Build or extend a golden set of 50 questions from real usage.
Run the harness above on three chunk sizes and the recursive and heading-aware splitters.
Only if you see lost-context failures (pronouns, unlabeled tables), try late chunking or contextual retrieval and compare cost per document.
Lock the winner into your ingestion config and add the benchmark to CI so embedding or parser changes get caught.

Chunking is unglamorous, but it is one of the few RAG decisions where an afternoon of measurement reliably pays back. If you want to work through retrieval evaluation and pipeline tuning hands-on with a cohort of engineers, take a look at the Joinloop AI Engineering cohort.