Building RAG: Chunking, HyDE and Caching

Vector Search to Reranking and Late Interaction built search: an index, keyword and dense scoring, fusion, reranking. Retrieval-augmented generation (RAG) puts a model on top: retrieve passages for the question, put them in the prompt, and have the model answer from them. (Lewis et al., 2020, who named it, trained the retriever and the generator together; today’s RAG keeps the idea and drops the joint training.) This post is about the parts between a search engine and a working RAG system: how the documents are cut up, what is embedded for the query, and what can be cached. They’re where most RAG systems fail, and the failure is silent: if the passage holding the answer is never retrieved, the model answers from whatever was, or from memory.

Table of contents

The pipeline

Indexing, once per document:

1. Clean the text: strip markup, keep the structure that carries meaning (headings, lists).
2. Chunk it into passages small enough to embed and to fit several in a prompt.
3. Embed each chunk, and store the vectors in an index (see Vector Search), usually beside a keyword index (see Semantic and Hybrid Search).

Answering, per question:

1. Embed the query, the question or something made from it.
2. Retrieve the top-k chunks, fuse and rerank them (see Semantic and Hybrid Search and Reranking and Late Interaction).
3. Generate: the chunks go into the prompt with instructions to answer from them and cite them.

Each step can lose the answer. This post measures the first two on this blog: given a question, does a chunk containing the answer come back?

The test: 150 questions with known answers

The test set comes from the blog itself. 150 sentences of prose are drawn at random from the posts, at most three per post; for each, Qwen2.5-3B writes a question the sentence answers, told to use its own words. The sentence is the gold answer: a retrieval hits if a returned chunk contains the whole sentence. That’s stricter than “found the right passage” (a chunk that cuts the sentence in half misses), and it’s the right strictness for RAG: half a sentence can say the opposite of the whole one.

common.py turns each post into clean text: no frontmatter, code, display math, components or tables, and links reduced to their text. It leaves out the posts that discuss this post’s experiments.

Full script: common.py

make_questions.py draws the sentences, then asks the model for a question, two rewordings of it and a hypothetical answer:

out = []
for slug, sent in items:
    q = ask("Write one question that the following sentence answers. Use your own words: don't copy its "
            f"distinctive phrases. Reply with the question only.\n\nSentence: {sent}")
    reworded = [ask(f"Reword this question {how}. Keep its meaning. Reply with the question only.\n\n{q}")
                for how in ("as a user typing quickly might", "more formally")]
    hyde = ask(f"Answer this question in two or three sentences, as a technical blog post would.\n\n{q}", 100)
    out.append({"post": slug, "sentence": sent, "question": q, "reworded": reworded, "hyde": hyde})

Full script: make_questions.py

The rewordings are for the semantic cache and the hypothetical answer is for HyDE, both used later. The 600 generations run on the GPU; on an Apple M4 laptop the whole run takes about 35 minutes.

Chunking

A chunk is the unit of retrieval: what gets embedded, scored and handed to the model. Its size is a trade. Too small, and a chunk lacks the context that makes it findable (“it” and “this method” match nothing) or useful. Too large, and its embedding averages several topics into a vector close to none of them; the model is handed text it doesn’t need; and the embedding model may not even read it all. MiniLM, used here, reads at most 256 word pieces, about 170 words of this blog’s text, and ignores the rest.

The strategies compared:

• Fixed size: every 200 words, regardless of content. Simple, and it cuts sentences and paragraphs anywhere.
• Fixed with overlap: windows of 200 words starting every 150, so a sentence cut by one boundary is whole in the next window. Costs about 30% more chunks.
• Recursive: split on the largest natural boundary that fits, here paragraphs merged up to 200 words, with an over-long paragraph split into sentences. The common default: LangChain’s documentation says of its recursive splitter “This text splitter is the recommended one for generic text”, and LlamaIndex’s SentenceSplitter “attempts to split text while respecting the boundaries of sentences” (both measure size in characters or tokens; here, words). Also run at up to 100 words, to see what size does, and within each section separately, as a control for the next one.
• By section: one chunk per heading. Respects the author’s structure; sections vary from a paragraph to thousands of words.
• Semantic: split between sentences where the topic shifts, measured by the embedding similarity of adjacent sentences (here, the lowest 20% of similarities), up to 200 words.
• Contextual: recursive chunks, each embedded with the post’s title and the section’s heading in front. Anthropic’s Contextual Retrieval has a model write 50–100 tokens of context per chunk instead, and reported that it cut retrieval failures (top 20) by 35%, and by 49% with the same context added to the BM25 index; title and heading are the free version, compared with the same chunks without them.
• Small-to-big: embed small chunks (60 words) for precise matching, but return the whole section a match comes from, for context.

The chunkers that need more than a split, from retrieval.py:

def windows(words, size, stride):
    return [" ".join(words[i:i + size]) for i in range(0, max(len(words) - size + stride, 1), stride)]
def recursive(text, size=200):
    """Paragraphs merged up to `size` words; a longer paragraph split by sentences."""
    out, cur = [], []
    for para in re.split(r"\n\s*\n", text):
        units = [para] if len(para.split()) <= size else sentences(para)
        for u in units:
            if cur and len(" ".join(cur).split()) + len(u.split()) > size:
                out.append(" ".join(cur)); cur = []
            cur.append(u)
    return out + ([" ".join(cur)] if cur else [])
def semantic(text, size=200):
    """Sentences grouped; a new chunk starts where adjacent sentences are least similar."""
    sents = sentences(" ".join(text.split()))
    if len(sents) < 2:
        return [" ".join(sents)]
    e = embed(sents); sim = (e[:-1] * e[1:]).sum(1); cut = np.quantile(sim, 0.2)
    out, cur = [], [sents[0]]
    for s, v in zip(sents[1:], sim):
        if v < cut or len(" ".join(cur + [s]).split()) > size:
            out.append(" ".join(cur)); cur = []
        cur.append(s)
    return out + [" ".join(cur)]

Full script: retrieval.py

The script embeds each strategy’s chunks with MiniLM, retrieves the top 1 and top 5 for each of the 150 questions, and counts the hits. A paired comparison uses an exact sign test on the questions where two strategies disagree. The script goes on to HyDE and the two caches, later in this post:

67 posts, 150 questions; MiniLM, dense retrieval

chunking                              chunks  words each  findable  hit@1  hit@5  words returned (top 5)  top 5 against recursive 200
fixed, 200 words                         870         192       133   52/150  83/150                     976  
fixed, 200 words, 50 overlap           1,124         195       149   51/150  93/150                     986  
recursive, up to 200 words             1,062         157       149   52/150  94/150                     785  
recursive, up to 100 words             2,065          81       149   67/150 103/150                     417  20 vs 11, p = 0.15
one chunk per section                    746         222       149   53/150  85/150                   1,137  16 vs 25, p = 0.21
semantic, up to 200 words              1,577         106       149   67/150 100/150                     607  21 vs 15, p = 0.41
recursive within each section          1,310         124       149   61/150 100/150                     658  15 vs 9, p = 0.31
contextual: title + heading added      1,310         124       149   59/150  98/150                     605  14 vs 10, p = 0.54
small-to-big: 60 words -> section      3,149         304       149   81/150 113/150                   1,267  27 vs 8, p = 0.00
fixed windows, overlap against none: 20 vs 10, p = 0.10; recursive 200 against fixed: 23 vs 12, p = 0.09
recursive 100 at top 10 (a word budget near small-to-big's): hit 114/150, 830 words returned
(MiniLM reads at most 256 word pieces, about 170 words of this blog; the rest of a longer chunk is not embedded)

HyDE, on recursive chunks: what is embedded as the query
  the question                     hit@1  52  hit@5  94  (vs the question at top 5: 0 gained, 0 lost)
  a hypothetical answer (HyDE)     hit@1  31  hit@5  73  (vs the question at top 5: 5 gained, 26 lost)
  question + answer, texts joined  hit@1  40  hit@5  82  (vs the question at top 5: 4 gained, 16 lost)
  question and answer embeddings averaged hit@1  43  hit@5  89  (vs the question at top 5: 3 gained, 8 lost)

embedding cache: 1,062 chunks embedded in 7.8 s cold, 0.001 s from the cache

semantic cache: 75 questions cached; 300 queries, 150 of them rewordings of a cached question
 threshold  repeats served  wrong answers served  of which to new queries
      0.70         137/150                 0/300                    0/150
      0.75         129/150                 0/300                    0/150
      0.80         115/150                 0/300                    0/150
      0.85          95/150                 0/300                    0/150
      0.90          72/150                 0/300                    0/150
      0.95          35/150                 0/300                    0/150

harder test: 114 pairs of different questions about the same post, against each question's own rewording
 threshold  same-post pairs above it (would be confused)   rewordings above it (hits)
      0.70                                     0/114                      125/150
      0.75                                     0/114                      115/150
      0.80                                     0/114                       97/150
      0.85                                     0/114                       79/150
      0.90                                     0/114                       59/150
      0.95                                     0/114                       32/150
  highest similarity between two different same-post questions: 0.66

hardest test: 12 hand-written pairs that differ by a word or two and need different answers
  similarities: 0.99, 0.98, 0.88, 0.86, 0.79, 0.59, 0.50, 0.48, 0.45, 0.43, 0.42, 0.41
  at threshold 0.80: 4 of 12 pairs would share one cached answer
  at threshold 0.85: 4 of 12 pairs would share one cached answer
  at threshold 0.90: 2 of 12 pairs would share one cached answer
  at threshold 0.95: 2 of 12 pairs would share one cached answer

First, how much to trust the table. The questions were written by a 3B model, and two hand checks of 15 at random disagreed on how many were fair: between 8 and 10 were fair questions the sentence answers, and the rest were too vague to point to that one sentence (“How can engine movement affect the behavior of a sampling setting?”) or misstated it. One answer sentence was also edited out of its post after the questions were made, so 149 is the most any strategy can find. Some also refer to context a reader wouldn’t have (“this post”). And with 150 questions, most of the differences below are within noise: the last column counts, against recursive 200-word chunks, the questions only one of the two strategies found in its top five, and an exact sign test on those. Only small-to-big’s lead is clearly real (p < 0.01). The rest are directions, not findings.

Cutting mid-sentence costs answers. Fixed 200-word windows left 16 of the 149 findable answer sentences split across two chunks, findable in no single chunk at all (the “findable” column). Overlap fixes that: windows that overlap by 50 words found 93 in the top five against 83, at about 30% more chunks. Recursive chunking, which never cuts a paragraph that fits, got 94 with no overlap and a fifth fewer words returned.

Smaller chunks did a little better. MiniLM reads about 170 words of this blog and drops the rest, and it was trained on texts of at most 128 word pieces, so a 200-word chunk is partly invisible to search and mostly longer than anything it learned on. Recursive chunks of up to 100 words found 103 in the top five against 94, returning half the words. One chunk per section, averaging 222 words, found 85.

The free context prefix didn’t help. Recursive chunks cut within each section, with nothing added, found 100. The same chunks with the post’s title and the section’s heading in front found 98. Anthropic’s version has a model write 50–100 tokens of context per chunk, about what the chunk means in its document, and reported a third fewer failures; a title and a heading carry less than that, and here they carried nothing measurable.

Semantic chunking matched the others. Splitting where adjacent sentences were least similar (here at the lowest 20% of similarities, a much more aggressive setting than the 95th-percentile breakpoint in LlamaIndex’s semantic splitter example, which cuts at about 5%) found 100, within noise of the recursive chunks.

Small-to-big found the most, by returning the most. Matching 60-word pieces and returning their whole section found the answer for 81 questions at the top and 113 in the top five, the only gain here that’s clearly not noise. But it returned 1,267 words per query, and a hit only needs the answer to be somewhere in a whole section, an easier test than the other strategies face. With about a third less text, recursive 100-word chunks at the top ten found 114 in 830 words. Small chunks for matching and more text for reading is the right shape; how much more is a cost decision.

HyDE: embed an answer, not the question

A question and its answer are different kinds of text: “why does X happen?” against “X happens because…”. HyDE (hypothetical document embeddings; Gao et al., 2022) bridges them by having a model write a plausible answer first, without retrieval, and embedding that instead. The hypothetical answer may be wrong in its details, but it’s written in the vocabulary and shape of the documents, and the nearest real passage to it is often the right one. The cost is a generation before every search, which adds a model call’s latency to every query.

The paper’s method is slightly different from the summary above: it samples several hypothetical documents and averages their embeddings with the question’s, and its encoder was Contriever, trained without any question–answer pairs, with InstructGPT writing the documents. It was tested on web search, BEIR’s low-resource sets and multilingual retrieval. Here, with one hypothetical answer of up to 100 tokens from the 3B model:

• embedding the question found the answer in the top five 94 times;
• embedding the hypothetical answer instead, 73 times: it gained 5 questions and lost 26;
• the two texts joined, 82;
• the two embeddings averaged, the paper’s way, 89.

HyDE lowered the hit rate however it was combined. Two reasons. MiniLM was trained on about a billion pairs, many of them questions with their answers, so it already maps a question near its answer, which is the gap HyDE exists to bridge; the paper itself found that with a fine-tuned encoder, HyDE helped less and a weaker generator made results worse. And the hypothetical answers were often wrong about this blog. Asked what kind of curve accuracy makes against the answer’s position, the model wrote that it “typically forms an S-shaped curve”; the post’s answer is a U. Asked how “engine movement” affects a sampling setting, it wrote about “the accuracy and reliability of the data collected.” HyDE needs measuring with the system’s own encoder and corpus before it is kept, and it adds a generation’s latency to every query.

Caching

Two caches, at two ends of the pipeline.

The embedding cache stores each chunk’s vector under a key that identifies both the text and the model: a hash of the model’s name and the chunk’s text. Re-indexing after an edit then embeds only the chunks that changed, and a new embedding model can’t silently reuse an old model’s vectors, because its keys differ:

# --- the embedding cache: key = hash(model name, text) ---
CACHE, MODEL = {}, "all-MiniLM-L6-v2"
def embed(texts):
    keys = [hashlib.sha256(f"{MODEL}\0{t}".encode()).hexdigest() for t in texts]
    todo = [t for t, k in zip(texts, keys) if k not in CACHE]
    for t, v in zip(todo, model.encode(todo, batch_size=64, normalize_embeddings=True) if todo else []):
        CACHE[hashlib.sha256(f"{MODEL}\0{t}".encode()).hexdigest()] = v
    return np.array([CACHE[k] for k in keys])

Full script: retrieval.py

retrieval.py times it: the whole index embedded from scratch, then again from the cache.

The semantic cache stores answers. Before answering a query, find the most similar earlier query by embedding; if the similarity clears a threshold, return that query’s answer without retrieval or generation. GPTCache is one implementation. The threshold decides everything: too low, and different questions get each other’s answers; too high, and a reworded repeat misses the cache. The run above measured both sides: 75 questions cached, then 300 queries, half of them rewordings of a cached question (which should hit, and get their own answer) and half new (which should miss):

cached = embed(questions[:75])
queries, own = [], []
for i, it in enumerate(items):
    for r in it["reworded"]:
        queries.append(r); own.append(i if i < 75 else -1)
own = np.array(own); S = embed(queries) @ cached.T; best, score = S.argmax(1), S.max(1)
print("\nsemantic cache: 75 questions cached; 300 queries, 150 of them rewordings of a cached question")
print(f"{'threshold':>10s} {'repeats served':>15s} {'wrong answers served':>21s} {'of which to new queries':>24s}")
for t in (0.70, 0.75, 0.80, 0.85, 0.90, 0.95):
    served = score >= t
    right = served & (best == own); wrong = served & (best != own)
    print(f"{t:>10.2f} {right.sum():>11d}/150 {wrong.sum():>17d}/300 {(wrong & (own < 0)).sum():>20d}/150")

Full script: retrieval.py

The embedding cache did what it should: the 1,062 recursive chunks took 7.8 seconds to embed and a thousandth of a second from the cache. In a real index it matters at re-indexing, when only edited chunks get new keys. The key must identify everything that shapes the vector: the model’s name and revision, any prefix the model expects (some want “query: ” or “passage: ” in front), and the truncation length, or a change to any of them silently reuses stale vectors.

The semantic cache needs more care. On the generated questions in the run above, no threshold served a wrong answer: rewordings scored high against their originals, and even different questions about the same post never scored above 0.66. A threshold of 0.80 served 115 of the 150 repeats; 0.90 only 72. A rerun on an NVIDIA RTX 3090 (its saved outputs), on a regenerated set of questions, served one wrong answer in 300 at 0.70 and none from 0.75 up, and its closest two same-post questions scored 0.70. On that evidence, 0.80 looks safe. But that test is easy, since the “new” questions are about unrelated sentences.

The hand-written pairs are the hard cases, chosen to show where similarity fails rather than to measure how often. They differ by a word or two and need different answers, and to the embedding model the hardest ones are nearly the same question:

• “When does speculative decoding help?” against “…not help?”: 0.99;
• “Why do keys and values get cached but not queries?” against the same with the roles swapped: 0.98;
• forward against reverse KL with a two-mode teacher: 0.88;
• BM25’s k1 against its b: 0.86.

At 0.80, 4 of the 12 pairs would share one cached answer, and even at 0.95 the negation and the swap would. Sentence embeddings capture topic, and negation and swapped roles barely change the topic. A threshold tuned on ordinary rewordings never meets these cases, and they’re the ones where a wrong answer looks right. A semantic cache is safe for questions whose answers don’t flip on a word, such as FAQ-style traffic, or behind a check that compares the new question with the cached one by more than embedding similarity, such as a cross-encoder (see Reranking and Late Interaction).

Grounding and citation

Retrieval decides what the model can see; the prompt decides what it does with it. Four habits make answers checkable:

• Answer from the context, or say it isn’t there. Instruct the model to answer only from the retrieved passages and to say so when they don’t contain the answer, instead of falling back on what it remembers.
• Cite. Number the passages and require each claim to name its passage, quoting the words it relies on. A citation can then be checked by a person, or by code that confirms the cited passage contains the quote; a paraphrase can’t be checked that way.
• Treat passages as data. Retrieved text is input from whoever wrote the documents, and it can carry instructions; fence it off in the prompt and tell the model not to follow it (see Guardrails and Prompt Injection).
• Log what was retrieved. Store the chunk ids with the answer; when an answer is wrong, the first question is whether the answer’s passage was retrieved at all, which is what this post measured.

Evaluating LLMs measures the answers themselves: metrics, error bars, and LLM judges, with their biases measured.

RAG, fine-tuning or long context?

Three ways to get knowledge into an answer:

• Long context: put everything in the prompt. Anthropic’s guidance is that a knowledge base under 200,000 tokens can simply be included, with prompt caching to make repeated use cheap. It loses in the middle of long inputs (Long Context, and Where It Breaks), and its cost grows with the corpus on every query.
• RAG: retrieve a few thousand tokens per question. Its per-query cost barely grows with the corpus, it reflects a document’s change as soon as it’s re-indexed, and it can cite its sources. It fails when retrieval does, which is what this post measured.
• Fine-tuning (Fine-Tuning, and What It Destroys): train the knowledge into the weights. Good for teaching a style, format or skill; poor for facts that change or must be cited, and it can damage what the model knew.

The usual answer for a body of documents is RAG, with long context when the documents are few, and fine-tuning for behavior rather than facts.

What a query costs

The run above counts the words a query returns: 417 with the five 100-word recursive chunks, 1,267 with small-to-big. A subword tokenizer gives at least one token per word, so at an illustrative $3 per million input tokens the passages cost at least $0.0013 or $0.0038 a query. At $15 per million output tokens, a 150-token answer costs $0.00225. Retrieval itself costs milliseconds: one query embedding and a search over a thousand vectors.

So the passages cost at least a little over half of the answer’s price, and up to at least about 1.7 times it, and the chunking choice sets their size. HyDE adds a generation to every query, about as expensive as the answer itself, and here it lowered the hit rate. A semantic cache hit costs one embedding and saves everything else, on the condition above. A query’s cost is set by choices made before the model runs.

Where this leads

• From Function Calling to the Agent Loop turns retrieval into a tool the model calls, and measures whether small models choose to search before answering.
• Beyond Vector RAG goes past one retrieval pass: agentic RAG, GraphRAG and vectorless retrieval.
• Evaluating LLMs measures the answers themselves: metrics, error bars, and LLM judges, with their biases measured.
• The Notebook Assistant, End to End builds the whole pipeline, with this post’s cache and chunks, and scores it on the same 150 questions.

Takeaways

• Most RAG failures happen before generation: the passage with the answer is never retrieved. Measure retrieval on the system’s own documents, by whether the answer comes back, before tuning the prompt, and test differences for significance: with 150 questions, most of this post’s are noise.
• Don’t cut answers in half: fixed windows split 16 of 149 answer sentences; overlap or paragraph-aware (recursive) chunking avoids it.
• Fit chunks to the embedding model: MiniLM reads about 170 words, and 100-word chunks did a little better than 200-word ones.
• Returning more text around a precise match finds more answers (small-to-big, 113 of 150), but a top-ten of small chunks found as many with a third less text.
• HyDE helps most with encoders not trained on question–answer pairs and a strong generator. With MiniLM and a 3B model it lowered top-five hits from 94 to 89 at best.
• Key an embedding cache by everything that shapes the vector. Tune a semantic cache on hard cases, not ordinary rewordings: “help” against “not help” scored 0.99.
• RAG suits a large or changing corpus with cited answers; long context suits a few documents; fine-tuning suits behavior, not facts.

References

Start with Anthropic’s Contextual Retrieval post for a practical, measured take on chunking, then Gao et al. for HyDE and Lewis et al. for where RAG began.

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020): the paper that named RAG, with a retriever and generator trained together.
• Gao et al., Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE, 2022): HyDE, its averaged embeddings, and its finding that a fine-tuned encoder gains less.
• Anthropic, Introducing Contextual Retrieval (2024): model-written context per chunk, and the 35% and 49% fewer retrieval failures it reported.
• Zilliz, GPTCache (repository, accessed October 2026): a semantic-cache implementation.
• Reimers and Gurevych, Sentence-BERT (2019), and the all-MiniLM-L6-v2 model card: the encoder used for every retrieval here.