Beyond Vector RAG: Agentic, Graph and Vectorless Retrieval

A retrieval pipeline of the kind Semantic and Hybrid Search and Reranking and Late Interaction built makes one pass: embed the question, fetch the nearest chunks, hand them to the model. That works when the answer sits in a few chunks that look like the question. It fails when the question needs several hops (“which post introduced the idea this one builds on?”), a view of the whole corpus (“what themes recur across the posts?”), or a document whose structure matters more than its words. This post covers three ways past one-pass retrieval.

Table of contents

Agentic RAG: retrieval in a loop

The simplest extension gives the model the search engine as a tool (From Function Calling to the Agent Loop) and lets it decide: search, read, decide whether it knows enough, search again with a better query, or answer. Multi-hop questions become a sequence of single-hop searches, each informed by the last. Self-RAG (Asai et al., 2023) trained a model to make these decisions explicitly, emitting tokens that say whether to retrieve, whether a passage is relevant, whether its own answer is supported, and how useful the answer is. The cost is the agent loop’s: several model calls per question, a growing context, and the risk of wandering. Coding agents (How Coding Agents Work) are agentic retrieval with grep as the search engine, and their move away from embedding indexes is evidence of how far a capable model can take it.

GraphRAG: retrieve structure

GraphRAG (Edge et al., 2024) builds a knowledge graph before any question is asked. An LLM reads every chunk and extracts entities (people, systems, concepts) and the relationships between them; the entities are merged across chunks into one graph; a community-detection algorithm (Leiden) groups densely connected entities into communities, at several levels of granularity; and an LLM writes a summary of each community. Queries then come in two kinds:

• Global search, the paper’s method, answers from one level’s community summaries, map-reduce style: the summaries are packed into context windows, each window yields a partial answer, and the partials are combined. This is what vector search can’t do: “what are the main themes of this corpus?” has no nearest chunk.
• Local search, added in Microsoft’s GraphRAG library, finds the entities whose descriptions are closest to the question by embedding, then gathers their relationships, communities and source text: good for questions about specific things. So GraphRAG still uses vectors; it adds a graph to them.

The price is paid mostly at indexing time, in LLM calls. For this blog, with the paper’s chunking and community-level ratios, and my assumptions for what the paper doesn’t state (listed in the opening comment of graphrag_cost.py):

chunks = -(-(total - 100) // 500)                     # 600-token windows advancing 500 at a time
PROMPT, OUT, SUMM_IN, SUMM_OUT = 1500, 500, 4000, 700
extract_calls = chunks * 3                            # extract, check, glean
extract_in = chunks * ((PROMPT + 600) + (PROMPT + 600 + OUT) + (PROMPT + 600 + OUT + 1))
extract_out = chunks * (OUT + 1 + OUT)
print(f"corpus: {total:,} tokens -> {chunks:,} chunks of 600 tokens")
print(f"vector index: {chunks:,} chunks embedded once, 0 LLM calls\n")
for ratio in (1.42, 1.61):
    summaries = round(chunks * ratio)
    tin, tout = extract_in + summaries * SUMM_IN, extract_out + summaries * SUMM_OUT
    print(f"GraphRAG at {ratio} summaries per chunk: {extract_calls:,} extraction calls + {summaries:,} "
          f"summaries = {extract_calls + summaries:,} LLM calls")
    print(f"  {tin / 1e6:.1f}M tokens in, {tout / 1e6:.1f}M out ({tin / total:.0f}x the corpus as input); "
          f"${tin / 1e6 * 0.10 + tout / 1e6 * 0.40:,.2f} at $0.10/$0.40 per M, "
          f"${tin / 1e6 * 3 + tout / 1e6 * 15:,.2f} at $3/$15 per M")
source = chunks * 600                                 # the chunked text, overlap included (Table 2's base)
print(f"\nper global-search query: {0.025 * source:,.0f} tokens of summaries at C0, "
      f"{0.70 * source:,.0f} at C3 (map calls of up to 8,000 tokens each, before prompts: "
      f"{-(-0.025 * source // 8000):.0f} and {-(-0.70 * source // 8000):.0f})")
print("per vector-search query: one embedding, then the top 5 chunks, about 3,000 tokens")

Full script: graphrag_cost.py

Before this, graphrag_cost.py counts the corpus with the Qwen2.5 tokenizer: every English post in the blog snapshot except this one and the next. The loop’s 1.42 and 1.61 are the community summaries per chunk, over all four levels, of the paper’s two corpora (its Table 2).

corpus: 557,546 tokens -> 1,115 chunks of 600 tokens
vector index: 1,115 chunks embedded once, 0 LLM calls

GraphRAG at 1.42 summaries per chunk: 3,345 extraction calls + 1,583 summaries = 4,928 LLM calls
  14.5M tokens in, 2.2M out (26x the corpus as input); $2.34 at $0.10/$0.40 per M, $76.78 at $3/$15 per M
GraphRAG at 1.61 summaries per chunk: 3,345 extraction calls + 1,795 summaries = 5,140 LLM calls
  15.3M tokens in, 2.4M out (27x the corpus as input); $2.48 at $0.10/$0.40 per M, $81.55 at $3/$15 per M

per global-search query: 16,725 tokens of summaries at C0, 468,300 at C3 (map calls of up to 8,000 tokens each, before prompts: 3 and 59)
per vector-search query: one embedding, then the top 5 chunks, about 3,000 tokens

About 5,000 LLM calls and 15 million input tokens, more than 25 times the corpus, against zero LLM calls for a vector index. It’s an estimate under assumptions the paper doesn’t fix, more likely low than high: it leaves out merging each entity’s descriptions and extracting claims, though it also runs a gleaning round on every chunk where the paper runs one only when a check says entities were missed. That’s a couple of dollars with a small model and around $80 with a frontier one for this half-million-token blog, and it scales with the corpus: a company wiki a thousand times larger is a thousand times the bill, and every substantial change means re-extracting what changed.

Global search also costs at query time. The paper’s root-level summaries came to about 2.5% of the chunked source text, so a question here reads about 17,000 tokens of them; its finest level came to about 70%, nearly half a million tokens and some 60 map calls per question, against about 3,000 tokens of retrieved chunks for vector search. GraphRAG earns those costs on questions about the corpus as a whole, and on relationship-heavy material; for questions a nearest chunk answers, it’s an expensive way to do what vector search does.

Vectorless retrieval: navigate the outline

Documents written by people have structure: titles, sections, headings. Vectorless or tree-index retrieval (PageIndex is one implementation) skips embeddings entirely and has an LLM navigate that structure the way a reader uses a table of contents: read the list of documents, pick one, read its outline, pick a section, read it. No index to build, no chunking that splits an argument in half, and every step is an explainable choice. The cost moves to query time, one LLM call per level, and everything depends on the outline being informative and the model choosing well.

Compared on this blog

Thirty questions, each answered by one known section of this blog, written by hand and mostly worded to avoid that section’s heading (otherwise heading navigation would win trivially). They cover 10 posts, three questions each, so the counts below are out of 30, and a gap of a few questions is within noise. Four kinds of method:

• keyword search (BM25), dense search (a small embedding model, scoring each section by its best 80-word passage), and the two fused, all three ranking every section of every post;
• vectorless navigation, Qwen2.5-3B-Instruct choosing from lists: a post from the list of titles, then a section from its headings. Three variants: bare titles; each title followed by the post’s one-paragraph description (tree-index tools give the model a summary per node for the same reason); and three levels, a topic first (the site’s 13, each listed with its posts’ titles), then a post within it, then a section. Each variant runs twice: replying with a number only, and reasoning briefly before the answer.

Two checks isolate where navigation breaks: the section choice alone, given the right post; and the post choice with the title list reversed, to see whether the picks follow position. On an Apple M4 laptop the whole comparison takes about 45 minutes, most of it the 3B model’s calls.

@torch.no_grad()
def pick(prompt, options, reason):
    """The LLM picks an option by number; None if the reply has no valid number."""
    listing = "\n".join(f"{i + 1}. {o}" for i, o in enumerate(options))
    ask = ("Think briefly about which option fits, then end with: Answer: <number>" if reason
           else "Reply with the number only.")
    ids = tok.apply_chat_template([{"role": "user", "content": f"{prompt}\n\n{listing}\n\n{ask}"}],
                                  add_generation_prompt=True, return_tensors="pt", return_dict=True)["input_ids"].to(DEVICE)
    out = lm.generate(ids, max_new_tokens=300 if reason else 6, do_sample=False, temperature=None, top_p=None, top_k=None)
    reply = tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True)
    m = re.search(r"Answer:\s*\**\s*(\d+)", reply) if reason else re.search(r"\d+", reply)
    k = int(m.group(1 if reason else 0)) - 1 if m else -1
    stats["calls"] += 1; stats["prompt"] += ids.shape[1]
    if not 0 <= k < len(options):
        stats["invalid"] += 1
        return None
    return k

slugs = list(tocs)
topics = sorted(set(topic[s] for s in slugs))
def section(q, slug, reason):
    h = pick(f"Question: {q}\nWhich section of the post '{tocs[slug][0]}' answers it?", tocs[slug][1], reason)
    return None if h is None else index[(slug, tocs[slug][1][h])]
def flat(q, reason, described=False, order=None):
    order = order or slugs
    p = pick(f"Question: {q}\nWhich blog post most likely answers it?",
             [tocs[s][0] + (f": {descs[s]}" if described else "") for s in order], reason)
    return (None, None, p) if p is None else (order[p], section(q, order[p], reason), p)
def three_levels(q, reason):
    t = pick(f"Question: {q}\nWhich topic's posts most likely answer it?",
             [f"{t}: " + "; ".join(tocs[s][0] for s in slugs if topic[s] == t) for t in topics], reason)
    if t is None:
        return None, None, None
    in_t = [s for s in slugs if topic[s] == topics[t]]
    p = pick(f"Question: {q}\nWhich blog post most likely answers it?", [f"{tocs[s][0]}: {descs[s]}" for s in in_t], reason)
    return (None, None, topics[t]) if p is None else (in_t[p], section(q, in_t[p], reason), topics[t])

Full script: compare.py

pick is the whole of vectorless navigation: one greedy call that chooses from a numbered list. flat and three_levels chain it from posts, or topics, down to a section. The rest of compare.py splits the posts into sections, holds the 30 questions, builds the BM25, dense and hybrid rankings, and prints the table and the two checks.

446 sections in 66 posts; 30 questions; counts are out of 30

method                                     right post  right section  section in top 5
keyword (BM25)                                     17             12                20
dense (MiniLM)                                     14             10                20
hybrid (fused)                                     20             12                24
vectorless, titles, number only                     2              1                 -
   59 calls, 1,374 prompt tokens per question, 1 invalid replies; picks by fifth of the list 2/0/7/2/18
vectorless, +descriptions, number only              0              0                 -
   60 calls, 8,084 prompt tokens per question, 3 invalid replies
vectorless, 3 levels, number only                   3              3                 -
   88 calls, 2,467 prompt tokens per question, 1 invalid replies; right topic 12 (of 13 topics), then right post 3 of those
   titles reversed: right post 0; picks by fifth of the reversed list 5/0/20/0/5
   section only, right post given: 18 (chance 4.1); 0 invalid replies
vectorless, titles, reasoning                       0              0                 -
   60 calls, 1,406 prompt tokens per question, 7 invalid replies; picks by fifth of the list 5/4/11/0/10
vectorless, +descriptions, reasoning                0              0                 -
   60 calls, 8,091 prompt tokens per question, 5 invalid replies
vectorless, 3 levels, reasoning                     3              2                 -
   79 calls, 2,146 prompt tokens per question, 15 invalid replies; right topic 6 (of 13 topics), then right post 3 of those
   titles reversed: right post 1; picks by fifth of the reversed list 14/3/10/0/3
   section only, right post given: 20 (chance 4.1); 1 invalid replies

questions where BM25 found the right post and dense search didn't:
   'What limits the speed of the plain attention implementation on': rarest shared words ['implementation', 'speed', 'gpu']
   'How does the preference loss follow from maximizing reward und': rarest shared words ['preference', 'kl', 'reward']
   "How is the output kept exactly distributed as the large model'": rarest shared words ['guesser', 'distributed', 'exactly']
   'How do two sequences share the same prompt memory until one of': rarest shared words ['share', 'sequences', 'prompt']
   "What maps a sequence's logical positions to physical memory ch": rarest shared words ['maps', 'logical', 'physical']
   'How is term frequency saturated and document length normalized': rarest shared words ['saturated', 'keyword', 'document']

Hybrid search did best: the right post first for 20 of 30 questions and the right section in the top five for 24, against 17 and 20 for keyword search alone. It tied keyword search on the top section (12 each). Keyword search beat dense search on the post (17 to 14). In about half of the six questions where it won, the last lines of the output show the question sharing a rare word with its section (“guesser,” “saturated,” “logical” and “physical”), which BM25 rewards heavily; in the rest the shared words were common ones, and the gap isn’t explained by this test.

LLM navigation barely worked: at most 3 right posts in 30, in any variant, either mode. Each variant is a chain of choices, and the checks show which link failed.

The section choice worked. Given the right post, the model picked the right section 18 times in 30 answering with a number, and 20 when it reasoned first, against about 4 by chance. A choice among a post’s six or seven headings is within its reach.

The post choice didn’t. From 66 titles it picked the right post 2 times in 30, and 18 of its 29 valid picks fell in the last fifth of the list, where none of the answers were. With the list reversed, it got none right, and 20 of its picks fell in the middle fifth. If particular titles attracted it, reversing the list would have moved those picks to the first fifth; if position alone did, they would have stayed at the end. Neither happened. The picks depend on the order, but not as a simple pull toward the end of the list, and they don’t track the questions: even when 20 picks landed in the middle fifth, which holds the answers to several of the questions, none was right. Adding each post’s description made the prompt six times longer (8,000 tokens) and the result worse, none right.

Smaller choices helped the first step only. With three levels, the model picked the right topic 12 times in 30 against about 2 by chance, but then the right post in only 3 of those 12, though each topic’s list held only a fraction of the 66 posts. The topics overlap (a post about KV caches could be “attention,” “inference-serving” or “efficiency”), so even the first step is ambiguous by construction. Reasoning before answering helped nowhere beyond noise, and it produced more replies with no valid number (15 of 79 calls in the three-level run).

What this measures is a 3B model navigating 66 documents, and it fails. PageIndex’s guidance points the same way: a basic model is enough to summarize the tree’s nodes, but the model that searches it should be “the best model you can afford.” Three differences from PageIndex limit how far this result reaches. Its open-source version navigates within one long document, where this test navigated across a corpus; it searches the tree as an agent, able to look and back up, where this test made single choices with no way back; and its developers’ reported 98.7% on FinanceBench is question answering over a single given filing, with the model unstated. I didn’t test a frontier model here. What the measurement does show is the shape of the cost: every query pays two or three LLM calls, 1,400 to 8,000 prompt tokens in all, where BM25 answers in milliseconds, and every choice in the chain must be right.

Choosing

• One-pass vector or hybrid search for questions answered by a passage that resembles them: the cheap default, and still the right one most of the time.
• Agentic retrieval when questions need several hops or the first query is likely to be wrong: pay in model calls per question.
• GraphRAG when questions are about the corpus as a whole or about relationships across documents: pay heavily at indexing time, and again per query for global search.
• Vectorless navigation for well-structured documents (manuals, filings, contracts) where the outline carries the meaning and every retrieval step should be explainable: pay a model call per level.

Where this leads

• Recursive Language Models drops retrieval as a separate step: the whole corpus sits in a REPL as a variable, and the model writes code to search it, scan it exhaustively, or call itself on pieces.
• Giving Agents Tools and Knowledge exposes a search engine like the ones above as an MCP tool, so any agent can do agentic retrieval over this blog.
• How Coding Agents Work looks at the agentic retrieval in production coding agents, which search with grep and file reads rather than an embedding index.

Takeaways

• Agentic RAG puts retrieval in the agent loop; coding agents show how far it goes with plain grep.
• GraphRAG extracts an entity graph and summarizes its communities, which answers whole-corpus questions vector search can’t; indexing this blog would take about 5,000 LLM calls, against none for a vector index.
• Vectorless retrieval navigates titles and headings with an LLM, moving the cost from indexing to every query.
• On this blog, hybrid search found the right post for 20 of 30 questions; LLM navigation with a 3B model found it for at most 3, with or without reasoning. The model chose well among one post’s headings (18 to 20 of 30) and failed among 66 posts; navigation needs a model that gets every choice in the tree right.

References

Start with the GraphRAG paper for the graph-and-communities pipeline and its global search, then Self-RAG for retrieval decisions a model makes itself, and the PageIndex README for tree navigation in practice. The Leiden paper is only needed for how the communities are found.

• Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023): a model trained to decide when to retrieve and to critique what it retrieved.
• Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (2024): the GraphRAG pipeline, global search, and the chunk and community ratios behind the cost estimate.
• VectifyAI, PageIndex (repository and README, accessed October 2026): vectorless tree navigation in practice, its model guidance and its FinanceBench claim.
• Traag, Waltman and van Eck, From Louvain to Leiden: guaranteeing well-connected communities (2019): the community-detection algorithm GraphRAG uses.