Chapter 10

RAG Systems · Chunking and retrieval strategies

📖 2 min read · 8 sections · May 2026

Why this matters

Embedding quality and DB choice get all the attention. Chunking strategy moves accuracy more than either. A bad chunking strategy will sink a great embedding model; a thoughtful one rescues a mediocre one.

Why chunking exists

Embedding models have a max input length (~8K tokens). You can't embed a 200-page PDF as one vector — even if you could, it would dilute the signal (the vector becomes "this is a 200-page document about everything", not useful). So you split into chunks, embed each, store each.

Chunking strategies, ordered by sophistication

Strategy	How	Trade-off
Fixed-size	Split every N characters/tokens	Simple; cuts mid-sentence
Fixed-size with overlap	N chars with M-char overlap between chunks	Mitigates mid-sentence cuts; more storage
Recursive	Split by paragraph, then by sentence if paragraph too big	Respects natural boundaries; LangChain default
Semantic	Embed sentences, cluster into chunks by similarity	Best for narrative text; expensive at index time
Document-aware	Use document structure: by section, table, code block	Best when docs have structure (markdown, HTML); needs parsing
Late chunking	Embed whole document with long-context model, then chunk the embedding output	Newest; preserves long-range context. Requires special model.

Chunk size — the real knob

Smaller chunks = more precise retrieval (less noise per match) but lose surrounding context. Larger chunks = more context but worse precision. Most apps land around 256-512 tokens with 10-20% overlap, but this is task-dependent. Always test.

Metadata is the cheap win

Store metadata with every chunk: source URL, page number, section, date, author, language. Filter on these aggressively at query time. Half the "RAG quality" wins teams claim from fancy retrieval come from "we filtered to the right docs first".

Hybrid retrieval (BM25 + dense)

Dense retrieval (vectors) catches semantic similarity. BM25 (keyword) catches exact matches and rare terms (acronyms, names, SKU codes). Both have failure modes; combined, they cover each other.

flowchart LR
    Q[User query] --> D[Dense search
top 20]
    Q --> B[BM25 search
top 20]
    D --> R[Reciprocal Rank Fusion]
    B --> R
    R --> RR[Re-ranker
cross-encoder]
    RR --> T[Top 5 to LLM]

Re-ranking — the second cheap win

After retrieving 20-50 candidates, run them through a small cross-encoder model that scores each (query, chunk) pair directly. Different model from the bi-encoder you used to index. Re-ranking improves precision dramatically — Cohere Rerank and BGE reranker are the popular choices.

Advanced tricks (when basics aren't enough)

HyDE (Hypothetical Document Embedding): generate a fake answer with an LLM, embed that, search. Useful when queries are short and underspecified.
Multi-query: rewrite the query 3-5 different ways with an LLM, search each, merge. Helps with vocabulary mismatch.
Self-querying: use the LLM to extract filters from the query ("show me errors from yesterday" → filter date=yesterday, then search "errors").
Parent-child chunks: index small chunks for precision; return the larger parent chunk for context to the LLM.

Q: My RAG returns the right chunks but the LLM still hallucinates. What do I check?

Three things, in order. (1) Are you sending the chunks in the prompt clearly labeled? "Here is context:" beats just pasting. (2) Are you instructing "answer ONLY from the context above; if not present, say 'I don't know'"? Without this, LLMs default to confident invention. (3) Is the context too long? Past ~8K tokens, LLMs start losing things in the middle. Cut to top-3 chunks.