DecisionRAG navigates 2.6M documents in under a second without a retriever
Researchers just introduced DecisionRAG, a new framework that ditches traditional vector retrievers in favor of decision models. Instead of slowly generating reasoning tokens, the model navigates documents like a library hierarchy, scoring entire branches in a single forward pass. It searches through 2.68 million Wikipedia passages in 0.7 to 1.8 seconds, beating dense retrievers with rerankers by double digits.
Anyone building RAG pipelines knows the retriever bottleneck: if the first stage fails to grab the right passage, even the smartest frontier model is helpless.
Teams often try using reasoning models to browse iteratively, but that crawls at a snail's pace. On October 26, 2026, researchers published DecisionRAG on arXiv, tossing out traditional vector retrievers entirely and replacing them with decision models.
Browsing like a human librarian
Traditional retrieval still leans heavily on lexical matching or rigid embedding spaces, leaving a huge capability gap between the retriever and the generator.
DecisionRAG takes a different route, mimicking how a person navigates a library. It organizes corpora into natural hierarchies, flowing down from broad topics and document titles straight to individual passages.
Instead of generating lengthy reasoning tokens word by word, the decision model evaluates all candidate branches at a given level in a single forward pass.
Because all candidates are scored within a shared context, their relevance estimates are directly comparable. The system simply follows the best branch down without wasting compute on text generation, running over 30 times faster than reasoning-based search agents.
Beating 8B rerankers in under a second
The benchmark numbers back up the approach. Testing across four benchmarks, DecisionRAG proved it can handle massive corpora without task-specific fine-tuning.
On a Wikipedia dataset packed with 2.68 million paragraphs, retrieval took just 0.7 to 1.8 seconds per query. The cost stayed between 0.2 and 2.3 cents per search.
Without needing a separate first-stage retriever, it handily outperformed dense retrieval paired with an 8B reranker. It beat that stack by 13.8 R@1 points on MultiDoc2Dial and by 17.4 Hit@1 points on Natural Questions.
Why it matters
We think DecisionRAG offers an elegant way out of the traditional retrieve-then-rerank trap.
For years, teams have thrown bigger rerankers at messy vector search results, ignoring the fact that if step one misses the chunk, step two is doomed. DecisionRAG brings the reasoning weight of language models right to the navigation step, but skips token generation to keep latency practical.
For developers managing well-organized document repositories, manuals, or structured knowledge bases, this architecture looks very promising. You get the judgment of an LLM at interactive speeds.
That said, the catch lies in the hierarchy. The system assumes a clean structural flow from categories to passages; applying this to messy, uncurated enterprise data will likely require a solid auto-structuring pipeline first.
Sources (1)
- arxiv.org(opens in a new tab)Originalarxiv.org