Your Embedding Model Isn't the Problem. Your Chunking Strategy Is.

The debate everyone has is the wrong one

Ask a team building a RAG system what they're optimizing and most will say "which embedding model." There's a real menu to pick from — OpenAI's text-embedding-3 line, Cohere's Embed v4, Voyage AI's voyage-3 family, Google's Gemini Embedding, and open-weight options like BGE-M3 and Alibaba's Qwen3-Embedding, which currently sits near the top of the MTEB multilingual leaderboard. That's a legitimate decision with real cost and latency tradeoffs attached.

It's also not where most retrieval quality gets lost. The MTEB leaderboard averages performance across roughly eight unrelated task types, and only a subset of those tasks resembles what your RAG system actually does — look up a specific passage given a specific question. A model ranked #1 overall can underperform a lower-ranked one on your corpus specifically. Rankings also move fast enough, and conflict across sources enough, that citing a frozen number is already out of date by the time you read it.

The decision that actually determines whether your retrieval works is upstream of any model: how you cut your documents into pieces before you embed them.

What naive chunking actually breaks

Split a document into fixed-size blocks — 500 characters, 1000 tokens, whatever — and you will, with total reliability, sever things that needed to stay together. A pronoun ends up in one chunk and the noun it refers to is in the previous one. A table's header row lands in chunk N while its data rows land in chunk N+1, so neither chunk means anything on its own. A qualifier established three paragraphs earlier ("this applies only to the enterprise tier") never travels with the sentence that depends on it.

The embedding model then does exactly what it's supposed to do: it encodes the chunk faithfully, including the ambiguity you just introduced. The resulting vector is a correct representation of an incomplete, sometimes misleading, piece of text. No amount of model quality fixes that, because the model was never shown the context that was lost.

This also shows up in document types that aren't prose. Code gets split mid-function, separating a function body from its signature. Tables get linearized incorrectly unless you explicitly repeat column headers on every row before chunking. Both require structure-aware splitting — chunking by AST node for code, by table boundary for tabular data — not a generic text splitter.

The fix that has real numbers behind it

Anthropic published a technique called Contextual Retrieval in 2024 that addresses this directly: before embedding a chunk, prepend a short, chunk-specific explanation of what it's about, generated by an LLM that has read the full document. A chunk that just says "the second phase begins six months after signing" becomes something like "This chunk is from the implementation timeline section of the Acme Corp contract; the second phase begins six months after signing." The embedding model now has what naive chunking took away.

Anthropic's own reported numbers, using their best-performing embedding configuration: plain chunking had a 5.7% top-20 retrieval failure rate. Contextual embeddings alone brought that to 3.7% (a 35% relative reduction). Adding a contextual BM25 keyword layer on top brought it to 2.9% (49%). Adding a reranking step on the retrieved candidates brought it to 1.9% — a 67% reduction from baseline. Worth saying plainly: that 67% figure is their best case, and per-domain results in their own writeup varied widely, from over 80% improvement in some domains to around 40% in others. Treat it as evidence the technique works, not a guarantee of a specific number on your data.

The LLM call per chunk has a real cost, though prompt caching keeps it manageable at indexing time (a one-time cost per chunk, not per query). If you want the same effect without an LLM call, "late chunking" — introduced by Jina AI and Weaviate researchers — runs a long-context embedding model over the entire document first, then splits the resulting token-level embeddings into chunks afterward, so each chunk's vector already reflects the whole document's context. Voyage AI's voyage-context-3 is a commercial version of the same idea: one embedding pass over the document, distinct vectors per chunk, document-level context baked into each one.

Fixed-size, recursive, or semantic — and why the obvious answer is wrong

Of the three common chunking strategies, semantic chunking — splitting at points where meaning shifts, rather than at a fixed size — sounds like it should always win. The evidence doesn't clearly say that. A study presented at NAACL 2025 by Vectara found that plain fixed-size chunking consistently outperformed semantic chunking across retrieval and answer-generation tasks. Separately, semantic chunkers tend to produce very small, uneven fragments — sometimes averaging well under 50 tokens — which can hurt even when the chunker correctly identifies a topic shift, because the fragment alone doesn't carry enough surrounding context to answer a question about it.

The strategy with the most consistent support across independent evaluations is recursive splitting: cut at paragraph breaks first, then line breaks, then spaces, falling back to fixed size only when nothing else is available — with roughly 256-512 tokens per chunk and 10-25% overlap as a starting point, then tuned against your own corpus and queries. Chunk size itself has a measurable effect independent of strategy: one evaluation on financial filings found hallucination rates were lowest around 1024-token chunks and noticeably higher at 128 tokens, which tracks with the general pattern — too small starves the model of context, too large buries the relevant sentence in noise.

Two failures that have nothing to do with chunking strategy

Even a well-chunked pipeline breaks in two specific ways that teams routinely miss. First, stale embeddings: most pipelines batch-embed on a schedule, so a document edited between runs is invisible to the index, and nothing errors — the system just serves outdated content with a plausible similarity score. Hashing source text and flagging mismatches on re-check is the standard fix. Second, embedding-model upgrades create mixed-generation indexes: vectors from the old model and the new model aren't on a comparable scale, so swapping models requires re-embedding the entire corpus, not just new documents going forward.

One practical upside worth knowing about: most current embedding models (OpenAI's text-embedding-3 line, Cohere Embed v4, Voyage, Qwen3-Embedding) are trained with Matryoshka representation learning, which front-loads the most important signal into the earliest dimensions of the vector. That means you can often truncate a 1536 or 3072-dimension vector down to 256 dimensions and keep most of the retrieval quality, cutting storage and search cost — a free lever most teams haven't pulled.

Takeaway: before you spend another cycle benchmarking embedding models against each other, audit how your documents are actually being split. If chunks lose the context a human reader would carry across sentences — a pronoun, a table header, a qualifier two paragraphs up — that's the failure mode to fix first, and it's cheaper to fix than switching models ever will be.