[Paper Review] SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMs

Jaehyung Kim, Jaehyun Nam+6 more|arXiv (Cornell University)|Apr 17, 2024

Natural Language Processing Techniques4 citations

TL;DR

SuRe proposes a zero-shot prompting framework that enhances open-domain question answering (ODQA) by generating candidate-conditioned summaries of retrieved passages to improve answer grounding and accuracy. By evaluating summary validity and relative informativeness, SuRe boosts F1 scores by up to 4.0% and exact match by up to 4.6% over standard prompting, with strong generalization across LLMs and retrieval methods.

ABSTRACT

Large language models (LLMs) have made significant advancements in various natural language processing tasks, including question answering (QA) tasks. While incorporating new information with the retrieval of relevant passages is a promising way to improve QA with LLMs, the existing methods often require additional fine-tuning which becomes infeasible with recent LLMs. Augmenting retrieved passages via prompting has the potential to address this limitation, but this direction has been limitedly explored. To this end, we design a simple yet effective framework to enhance open-domain QA (ODQA) with LLMs, based on the summarized retrieval (SuRe). SuRe helps LLMs predict more accurate answers for a given question, which are well-supported by the summarized retrieval that could be viewed as an explicit rationale extracted from the retrieved passages. Specifically, SuRe first constructs summaries of the retrieved passages for each of the multiple answer candidates. Then, SuRe confirms the most plausible answer from the candidate set by evaluating the validity and ranking of the generated summaries. Experimental results on diverse ODQA benchmarks demonstrate the superiority of SuRe, with improvements of up to 4.6% in exact match (EM) and 4.0% in F1 score over standard prompting approaches. SuRe also can be integrated with a broad range of retrieval methods and LLMs. Finally, the generated summaries from SuRe show additional advantages to measure the importance of retrieved passages and serve as more preferred rationales by models and humans.

Motivation & Objective

• To address the limitations of standard prompting in retrieval-augmented LLMs, where LLMs fail to effectively utilize long context and retrieve relevant information.
• To improve answer grounding and reduce hallucinations in open-domain QA by generating rationale-like summaries conditioned on answer candidates.
• To develop a zero-shot, plug-and-play framework that works with black-box LLM APIs and various retrieval methods without fine-tuning.
• To evaluate whether the generated summaries serve as better rationales for model and human interpretation.

Proposed method

• SuRe constructs multiple summaries of retrieved passages, each conditioned on a different answer candidate to focus on relevant context.
• It evaluates the validity of each candidate-summary pair using a large language model to assess whether the summary supports the candidate.
• It ranks candidate summaries by their informativeness in answering the question using pairwise comparison via LLMs.
• The final answer is selected based on the highest-ranked and most valid summary, forming a rationale for the prediction.
• All steps are performed via zero-shot prompting, requiring no fine-tuning or few-shot examples.
• The method is compatible with diverse retrieval methods (e.g., BM25, DPR, Contriever) and LLMs (e.g., GPT-4, ChatGPT, LLaMA-2).

Experimental results

Research questions

• RQ1 Can candidate-conditioned summarization improve the accuracy of zero-shot open-domain question answering with LLMs?
• RQ2 Can summarization generated via zero-shot prompting serve as a more reliable and interpretable rationale than generic summarization?
• RQ3 How does SuRe perform across diverse retrieval methods and LLMs without fine-tuning?
• RQ4 Can the generated summaries be used to measure the importance of retrieved passages?
• RQ5 Does SuRe improve model and human preference for rationales in QA systems?

Key findings

• SuRe improves exact match (EM) by up to 4.6% and F1 score by up to 4.0% over standard prompting on four ODQA benchmarks.
• With ChatGPT and BM25, SuRe achieves a 12.8% zero-shot QA accuracy gain compared to no retrieval augmentation.
• The method generalizes well across different LLMs (e.g., GPT-4, LLaMA-2, ChatGPT) and retrieval methods (BM25, DPR, Contriever).
• Generated summaries from SuRe are rated as more informative and preferred by both models and humans compared to generic summarization.
• The summaries effectively identify and rank the most relevant retrieved passages, enabling interpretability and importance assessment.
• SuRe outperforms strong baselines like chain-of-thought, self-verification, and RePlug across all datasets, with consistent gains in EM and F1.