[Paper Review] DR-RAG: Applying Dynamic Document Relevance to Retrieval-Augmented Generation for Question-Answering

Zijian Hei, Weiling Liu+6 more|arXiv (Cornell University)|Jun 11, 2024

Topic Modeling4 citations

TL;DR

DR-RAG proposes a two-stage retrieval-augmented generation framework that improves answer accuracy in multi-hop question-answering by dynamically identifying both highly relevant (static) and low-relevance but critical (dynamic) documents. By using a compact classifier to filter and select documents based on query-document relevance, DR-RAG enhances retrieval recall and answer precision while calling the LLM only once, achieving state-of-the-art results on multi-hop QA benchmarks.

ABSTRACT

Retrieval-Augmented Generation (RAG) has recently demonstrated the performance of Large Language Models (LLMs) in the knowledge-intensive tasks such as Question-Answering (QA). RAG expands the query context by incorporating external knowledge bases to enhance the response accuracy. However, it would be inefficient to access LLMs multiple times for each query and unreliable to retrieve all the relevant documents by a single query. We have found that even though there is low relevance between some critical documents and query, it is possible to retrieve the remaining documents by combining parts of the documents with the query. To mine the relevance, a two-stage retrieval framework called Dynamic-Relevant Retrieval-Augmented Generation (DR-RAG) is proposed to improve document retrieval recall and the accuracy of answers while maintaining efficiency. Additionally, a compact classifier is applied to two different selection strategies to determine the contribution of the retrieved documents to answering the query and retrieve the relatively relevant documents. Meanwhile, DR-RAG call the LLMs only once, which significantly improves the efficiency of the experiment. The experimental results on multi-hop QA datasets show that DR-RAG can significantly improve the accuracy of the answers and achieve new progress in QA systems.

Motivation & Objective

• To address the limitation of existing RAG systems in retrieving low-relevance but critical documents for multi-hop question-answering.
• To improve retrieval recall and answer accuracy by distinguishing between static-relevant and dynamic-relevant documents.
• To reduce inefficiency and hallucination by minimizing redundant or irrelevant document retrieval.
• To maintain high efficiency by calling the LLM only once per query through a two-stage retrieval and classifier-based selection strategy.

Proposed method

• A two-stage retrieval framework: first, similarity matching retrieves an initial set of documents based on query embedding.
• Second, query-document concatenation is used to uncover deeper relevance, especially for dynamic-relevant documents.
• A compact classifier evaluates document relevance using a predefined threshold to determine contribution to answer generation.
• Two selection strategies—forward and reverse selection—are applied to optimize the retrieved document set and reduce redundancy.
• The final document set is fed into the LLM only once, ensuring high inference efficiency.
• The framework integrates dynamic relevance mining into RAG to improve performance on complex, multi-hop QA tasks.

Experimental results

Research questions

• RQ1 Can a two-stage retrieval approach improve recall of critical but low-relevance documents in multi-hop QA?
• RQ2 How can dynamic relevance between queries and documents be effectively measured and leveraged in RAG?
• RQ3 To what extent can a lightweight classifier improve document selection without increasing inference cost?
• RQ4 Does reducing redundant document retrieval enhance answer accuracy in LLM-based QA systems?
• RQ5 Can a single LLM call achieve state-of-the-art performance when combined with intelligent document retrieval and filtering?

Key findings

• DR-RAG significantly improves answer accuracy on multi-hop QA datasets by effectively retrieving dynamic-relevant documents that are often missed by standard RAG.
• The method achieves higher retrieval recall compared to baseline RAG and query-dependent retrieval (QDC) methods, particularly for complex, multi-hop questions.
• The use of a compact classifier with forward and reverse selection strategies reduces irrelevant document inclusion, improving answer quality.
• By calling the LLM only once, DR-RAG maintains high inference efficiency while outperforming multi-call RAG baselines.
• In case studies on HotpotQA, DR-RAG successfully retrieves the necessary documents (e.g., Caroline LeRoy’s spouse) that other methods fail to identify due to low direct relevance.
• The framework demonstrates state-of-the-art performance on multi-hop QA benchmarks, showing new progress in LLM-based question-answering systems.