[Paper Review] Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval

Omar Khattab, Christopher Potts+1 more|arXiv (Cornell University)|Jan 2, 2021

Topic Modeling24 references4 citations

TL;DR

Baleen proposes a scalable, robust multi-hop reasoning system that improves retrieval accuracy in open-domain question answering and claim verification by using condensed retrieval to reduce search space, a focused late interaction retriever (FLIPR) for complex query modeling, and latent hop ordering to learn effective hop sequences from weak supervision. It achieves state-of-the-art performance on both two-hop (HotPotQA) and many-hop (HoVer) benchmarks.

ABSTRACT

Multi-hop reasoning (i.e., reasoning across two or more documents) is a key ingredient for NLP models that leverage large corpora to exhibit broad knowledge. To retrieve evidence passages, multi-hop models must contend with a fast-growing search space across the hops, represent complex queries that combine multiple information needs, and resolve ambiguity about the best order in which to hop between training passages. We tackle these problems via Baleen, a system that improves the accuracy of multi-hop retrieval while learning robustly from weak training signals in the many-hop setting. To tame the search space, we propose condensed retrieval, a pipeline that summarizes the retrieved passages after each hop into a single compact context. To model complex queries, we introduce a focused late interaction retriever that allows different parts of the same query representation to match disparate relevant passages. Lastly, to infer the hopping dependencies among unordered training passages, we devise latent hop ordering, a weak-supervision strategy in which the trained retriever itself selects the sequence of hops. We evaluate Baleen on retrieval for two-hop question answering and many-hop claim verification, establishing state-of-the-art performance.

Motivation & Objective

• To address the challenges of scaling multi-hop reasoning to many hops while maintaining high retrieval accuracy.
• To reduce the exponential search space inherent in multi-hop retrieval through context condensation after each hop.
• To model complex, multi-faceted queries that require matching disparate relevant passages using a flexible retrieval mechanism.
• To learn effective hop ordering without relying on brittle, dataset-specific heuristics or expensive manual annotations.
• To enable robust training from weak supervision signals in large-scale, many-hop retrieval tasks.

Proposed method

• Condensed retrieval: Summarizes retrieved passages after each hop into a compact context, which becomes part of the query for the next hop, reducing search space and improving scalability.
• Focused Late Interaction Passage Retriever (FLIPR): A learned retrieval model that allows different parts of the same query embedding to match different relevant passages, enabling fine-grained, context-aware matching.
• Latent hop ordering (LHO): A weak-supervision strategy where the retriever itself selects the optimal sequence of hops during training, avoiding reliance on manual or heuristic ordering.
• Hybrid architecture: Combines greedy retrieval with condensed context to balance recall and efficiency, outperforming beam search in scalability and performance.
• End-to-end training with weak supervision: Uses only passage-level labels (not hop-order annotations) to train the system, enabling generalization to arbitrary numbers of hops.
• Adaptation to large-scale corpora: Designed to scale efficiently to many hops and large document collections, leveraging the efficiency of late interaction models like ColBERT.

Experimental results

Research questions

• RQ1 Can a retrieval system achieve high recall in multi-hop reasoning tasks while scaling effectively to many hops?
• RQ2 How can complex queries requiring information from multiple, disparate passages be modeled more effectively than with standard vector or bag-of-words approaches?
• RQ3 Can effective hop ordering be learned automatically from weak supervision signals without explicit ordering annotations?
• RQ4 To what extent does condensed retrieval improve recall and robustness compared to standard greedy or beam search strategies?
• RQ5 How does the system perform on real-world, many-hop claim verification tasks beyond two-hop benchmarks?

Key findings

• Baleen achieves 96.3% answer recall in the top-20 retrieved passages on the two-hop HotPotQA benchmark, significantly outperforming prior work (89.4%).
• On the four-hop HoVer claim verification dataset, Baleen achieves 92.2% overall retrieval accuracy, substantially exceeding the next strongest baseline (74.8%).
• The ablation study shows that removing FLIPR reduces performance to 87.4% on HoVer, highlighting its critical role in modeling complex queries.
• Removing latent hop ordering (LHO) reduces performance to 84.8%, demonstrating that LHO is essential for effective hop sequence learning.
• The hybrid architecture (Baleen${}_{\textnormal{HYBRID}}$) achieves 94.5% accuracy on HoVer, showing that combining condensed retrieval with reranking improves robustness.
• Baleen's performance remains strong even at four hops (85.1% on HotPotQA), indicating scalability to complex, multi-hop reasoning tasks.