Reasoning-Based Retrieval for Multi-Hop Question Answering: ~92% Hits@10 on MultiHop-RAG, and Why the Rest Is Temporal Reasoning

Raveesh Gupta, Ronn Chinowutthichai Tableside / abotcompany

Public technical brief. Headline figures are from the full 2,255-query MultiHop-RAG dev-set run (91.7% Hits@10), scored with the dataset's own metric and logged per-query. This is a condensed version; full configuration and ablation logs are maintained internally.

---

Abstract

Retrieval-augmented generation (RAG) degrades sharply on multi-hop questions whose answer requires combining evidence spread across several documents. We present PageIndex retrieval, a reasoning-based pipeline that demotes the vector index to a coarse document filter and then lets a language model navigate each document's reconstructed structure to select the answering passages, followed by a cross-encoder rerank with a recall guarantee. On the MultiHop-RAG benchmark (Tang & Yang, 2024), under the dataset's own retrieval metric, PageIndex retrieval reaches 91.7% Hits@10 on the full 2,255-query dev set, versus a BM25 lexical floor at 76.9% and a prior dense-only configuration at 85.8%. More importantly, our error analysis overturns the intuitive assumption that misses scale with hop count: miss rate is inverse to the number of gold documents, and the entire performance gap is concentrated in temporal-reasoning queries. We argue the path past ~92% is reasoning-aware retrieval, not better ranking.

---

1. Introduction

A restaurant guest asking "is the tandoori roti gluten-free and under £4?" poses a two-hop question: one fact (allergens) and another (price) may live in different sections or documents, and the answer requires both. Standard RAG — embed the query, retrieve top-k by cosine similarity, generate — frequently retrieves passages that are topically on-target but miss one of the required hops. MultiHop-RAG (Tang & Yang, 2024) was built to measure exactly this, and reported retrievers leave a large headroom.

We make two contributions:

1. A method. PageIndex retrieval: a probe → document-selection → per-document structure navigation → rerank-with-recall-net pipeline that reaches 91.7% Hits@10 on the full MultiHop-RAG dev set.
2. An error analysis that reframes the problem. Multi-hop retrieval failure is temporal-reasoning-bound, not complexity-bound — to our knowledge, the first such decomposition on this benchmark, and it has direct design implications.

2. Method (overview)

PageIndex retrieval is a staged pipeline. Each document is ingested into a structure tree whose leaves preserve verbatim text.

• Probe — coarse document filter. A hybrid (dense + lexical) search returns candidate chunks; we keep their source documents. The vector index is used only to decide which documents are relevant, not which passages.
• Document selection. A cross-encoder rescores the candidate documents' content, so a document the probe ranked low can still advance to navigation.
• Structure navigation. For each selected document, a language model reads the document's reconstructed outline and selects the answering nodes — conditioned on evidence already gathered from earlier documents this turn, so it can pick the bridging passage a stateless pass would miss when the answer is split across documents.
• Final rerank with a recall net. The selected passages are reranked to the final top-K, with a safety net that protects each document's strongest passage from being dropped.

Decoding for the navigation step is deterministic. The pipeline is fail-soft: any navigation error falls back to the probe passages, so PageIndex is never worse than a standard retriever. (Exact configuration and model choices are maintained internally.)

3. Experimental Setup

Dataset. MultiHop-RAG (Tang & Yang, 2024): 2,556 queries over 609 news articles, each labeled with 2–4 gold evidence facts across reasoning types (inference, comparison, temporal, null). null queries (correct answer = "no evidence") are excluded from the retrieval aggregate by the paper's convention, leaving 2,255 scorable queries.

Metric. We port the dataset's official scoring exactly: a retrieved chunk is relevant iff a gold fact string is a substring of it. Hits@K is 1 if any top-K chunk contains any gold fact; we also track Hits@4, MAP@10, and MRR@10. We verified that 100% of gold facts survive verbatim in our ingested chunks, so the metric measures retrieval, not ingestion loss.

Provider equivalence. The navigation model was served across two providers over the course of the full run; we verified 100% per-query agreement on a matched subset under deterministic decoding, so the mixed-provider run is valid.

4. Results

On the full 2,255-query scorable set, PageIndex retrieval reaches 91.7% Hits@10 — +5.9 pp over the prior 85.8% configuration and +14.8 pp over the BM25 floor.

Per-reasoning-type (full set): inference 95.8%, comparison 92.9%, temporal 84.0%.

A deterministic navigation step and a final cross-encoder reranking stage with a recall net were the load-bearing improvements over the prior configuration; a learned-sparse lexical channel in the probe adds a further small gain, concentrated where exact term overlap matters. (Exact ablation configurations are omitted in this public version.)

5. Error Analysis: the gap is temporal reasoning, not hop count

This is the brief's central empirical finding.

(a) Failure is inverse to multi-hop complexity. Miss rate does not rise with the number of gold documents — it falls:

Under a lenient any-fact metric, more gold documents means more chances to surface one — so the "hardest" multi-hop questions are not where the system fails.

(b) The entire gap is temporal reasoning. Temporal queries score 84.0% versus 95.8% (inference) and 92.9% (comparison). Linguistic-marker analysis confirms it: comparison vocabulary is under-represented in misses (easier), while temporal markers are over-represented (harder). The worst sub-type is consistency-across-time — questions about whether a later report changed or contradicted an earlier one.

(c) The misses are stable, not a tuning artifact. Across distinct configurations, the large majority of misses occur in both — they are not recoverable by ranking tweaks. The lever past ~92% is reasoning-aware retrieval, not better fusion or ranking.

6. Limitations

(1) We measure retrieval Hits@K, not end-to-end answer accuracy. (2) Single corpus (news); restaurant/menu generalization is asserted by analogy, not yet measured. (3) The full-set run mixes two serving providers for the same open model (shown equivalent under deterministic decoding, but not identical to a frozen checkpoint). (4) The substring metric rewards verbatim recall and is insensitive to paraphrase.

Reproducibility & contact

Per-query outcomes and aggregate metrics are logged internally; the metric is the dataset's official substring scorer (null-excluded). An extended version of this work — with full configuration, model choices, and per-experiment ablations — is maintained internally and available to partners under NDA.

References

Tang & Yang (2024), MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries.