Indexing for RAG
Chunking, contextual retrieval, and how the library is arranged
Cutting the library into shelves
A reader’s guide to part of CS 329Z Lecture 3 (Diyi Yang, Michael Ryan and John Yang; Engineering AI Agents, Stanford, autumn 2026). The outline and the pointers come from that lecture; the wording on this page is ours. The course is also listed under Resources.
Retrieval is only as honest as the pieces you stored. If you cut a document in the wrong place, the best retriever in the world will still hand the model a fragment that does not mean what it should. Indexing is that unglamorous work: how you chunk, how you give a chunk its context, how you embed it, and how you organise the shelves so a query can find it quickly.
Think of photocopying a handbook into index cards. Cut too short and a card says “due Thursday” with no idea which Thursday. Cut too long and one card mixes grading, office hours, and the late-day policy, so its “topic” is a blur. Write a one-line header on each card — “From the grading section of the autumn syllabus” — and the same sentence becomes findable. That header is contextual retrieval. A tree of summary cards, or a map of who relates to whom, is what you reach for when the answer is not on a single card.
Embeddings, in one table
A retriever that uses vectors needs two things: a fixed-dimension representation of each chunk, and a similarity (usually cosine or an inner product). The lecture’s landscape is enough to steer by; we will not retell the history of word2vec here.
| Type | Examples | One vector covers |
|---|---|---|
| Static word | word2vec, GloVe | a word type |
| Contextual | BERT, RoBERTa | a token in context |
| Passage | Sentence-BERT, DPR, E5 | a chunk |
| Multimodal | CLIP | text or image, in a shared space |
| Graph | node2vec, GNN encoders | a node or a relation |
Passage embeddings are the usual unit for RAG: one vector per chunk, compared to a query vector. How you cut the chunk matters as much as which encoder you pick.
Chunking
Fixed-size chunking cuts on a token budget, often in the 100–500 range. Too small, and the vector has no context. Too large, and one vector averages several topics. A 10–20% overlap is a common hedge, so a sentence is less likely to be split at the cut. Semantic or structure-aware chunking follows headings, sections, or sentences instead of a raw counter. You can also carry metadata or a short summary with each chunk, or ask a model to find natural breakpoints. The lecture points at a Databricks guide to chunking strategies for a practical tour.
The same short grading paragraph, split three ways, is the picture to keep:
A mid-sentence cut leaves “11:59pm” on a card that no longer says what is due. Overlap saves the join. A heading-aware split keeps “Grading — Milestone” with the date.
Contextual retrieval
Even a clean chunk can be meaningless on its own. “The company’s revenue grew by 3% over the previous quarter” does not say which company or which quarter. Anthropic’s contextual retrieval prepends a short situating line, written by a model from the whole document, before the chunk is embedded and indexed. The query “What was the revenue growth for ACME Corp in Q2 2023?” can then match the card.
The method is a prompt over the whole document and the chunk: ask for a short context that situates the chunk for search, and store that prefix with the text. Anthropic also combines the contextual embedding with a contextual BM25 index, so meaning and exact terms can vote together — a hybrid we will meet again on the retrievers page.
On their evaluation, as shown in the lecture, average failed retrievals at 20 fall from 5.7% (embedding only) and 5.0% (embedding plus BM25) to 3.7% with contextual embeddings and 2.9% with contextual embeddings plus contextual BM25. Those are Anthropic’s reported failure rates, not a rerun of our own. The relative drop from 5.7% to 2.9% is the “about 49%” reduction that the blog highlights.
Late chunking (Günther et al., 2024) is a different way to give a chunk document context. You embed the whole document with a long-context encoder, then mean-pool the token vectors that fall inside each chunk’s span. There is no extra language-model call and no added prefix on the stored text: the context lives in the vector.
When one chunk is not enough
Some questions are about a theme, not a sentence. RAPTOR — Recursive Abstractive Processing for Tree-Organized Retrieval (Sarthi et al.) — builds a tree from the bottom up: embed the chunks, cluster them, summarise each cluster with a language model, embed the summaries, and repeat. A detail question can match a leaf; a thematic question can match a node. The lecture labels this Sarthi et al., 2025; the paper appeared in 2024 (arXiv, January 2024; ICLR 2024).
GraphRAG (Edge et al., 2024) is a cousin for questions that need a global view of a corpus. At index time an LLM extracts entities and relations, builds a knowledge graph, and detects communities. At query time it writes community summaries and then a global answer. Later surveys of graph-based RAG, such as Han et al. (2025), collect the many variants of that idea.
Finding a vector quickly
Once every chunk is a vector, the query is a nearest-neighbour search: given a query vector, return the top-\(k\) of \(N\) chunk vectors. An exact scan costs \(O(N \cdot d)\) per query. Approximate nearest neighbour (ANN) methods trade a little recall for a lot of speed. The lecture’s short list is inverted-file indexes (cluster, then probe nearby centroids), product quantisation (compress each vector into a short code), and HNSW — hierarchical navigable small worlds (Malkov and Yashunin).
HNSW is a multi-layer proximity graph: every vector lives on the bottom layer, thinner samples live above. Search starts at the top, hops greedily toward the query, and drops a layer when it is stuck. In the usual picture you pay about \(\log N\) hops and keep high recall. The index is built by inserting one vector at a time, placing most nodes only on the bottom layer, and linking each to a few neighbours plus a longer-range edge so a greedy walk cannot get trapped.
In one table
| Choice | What it is for |
|---|---|
| Fixed-size chunks, with overlap | A simple default; overlap reduces mid-sentence cuts |
| Structure-aware cuts | Keep headings and sections together |
| Contextual retrieval | A short preface, so a chunk names its document and moment |
| Late chunking | Pool a long-context embedding over each chunk’s span |
| RAPTOR | A tree of summaries, for questions that span many chunks |
| GraphRAG | A graph of entities and communities, for corpus-level questions |
| HNSW (and other ANN) | Fast approximate search over a large vector index |
FAQ
Is there one right chunk size? No. The lecture’s 100–500 token band is a starting range, not a law. Too short starves the vector; too long mixes topics. You measure, as you would any other index choice.
Do I need an LLM call for every chunk? Only if you use a method like contextual retrieval. Late chunking gets document context from the encoder itself. A heading and a filename, stored as metadata, are sometimes enough.
Why not embed the whole document as one vector? Because a single vector is a blur, and because most questions match a passage, not a book. Trees and graphs are there for the questions that do need a wider view.
Where does this meet retrieval? Indexing decides what a “passage” is. Retrievers decide how a query finds it: sparse overlap, a dense vector, or something in between.
See also
- Why RAG
- Retrievers for RAG
- Agentic RAG
- Failure modes of RAG
- LLMs for agents, for what happens once a passage reaches the context window
- RAG and agents: Resources