[Paper Review] Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

Bowen Jin, Jinsung Yoon+2 more|arXiv (Cornell University)|Oct 8, 2024

Scientific Computing and Data ManagementDecision Sciences3 citations

TL;DR

This paper investigates performance degradation in long-context retrieval-augmented generation (RAG) systems when increasing retrieved passage counts, identifying 'hard negatives' as a key cause. It proposes retrieval reordering (training-free) and two RAG-specific fine-tuning methods—implicit robustness tuning and explicit reasoning-based fine-tuning—demonstrating significant performance gains on multiple benchmarks, with the best method improving accuracy by up to 12.7% over base models.

ABSTRACT

Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs. It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance. However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases. This paper investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" as a key contributor. To mitigate this and enhance the robustness of long-context LLM-based RAG, we propose both training-free and training-based approaches. We first showcase the effectiveness of retrieval reordering as a simple yet powerful training-free optimization. Furthermore, we explore training-based methods, specifically RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning with intermediate reasoning, demonstrating their capacity for substantial performance gains. Finally, we conduct a systematic analysis of design choices for these training-based methods, including data distribution, retriever selection, and training context length.

Motivation & Objective

• To investigate why performance in long-context RAG systems degrades when increasing the number of retrieved passages despite higher recall.
• To identify 'hard negatives' in retrieved passages as a primary cause of performance decline in long-context LLMs.
• To develop training-free and training-based methods that enhance robustness to noisy and irrelevant retrieved content.
• To evaluate the impact of design choices such as data distribution, retriever selection, and context length on RAG-specific fine-tuning.
• To demonstrate that combining RAG-specific fine-tuning with general SFT data preserves general capabilities while boosting RAG performance.

Proposed method

• Retrieval reordering: reorganizes retrieved passages by placing high-scoring documents at the beginning and end of the input sequence to mitigate the 'lost-in-the-middle' attention issue.
• Implicit robustness fine-tuning: fine-tunes LLMs on query-passage pairs including noisy or hard-negative passages to improve general robustness without explicit relevance labeling.
• Explicit relevance fine-tuning: introduces an intermediate reasoning step during fine-tuning where the LLM identifies relevant information before generation, enhancing relevance discrimination.
• Training data scaling: evaluates performance across 5k to 200k samples of RAG-specific data, showing consistent accuracy gains with larger data scales.
• Hybrid fine-tuning: combines RAG-specific data with general SFT data (e.g., UltraChat) to maintain general capabilities while boosting RAG performance.
• Systematic ablation: evaluates the impact of retriever strength, training context length, and data distribution on fine-tuning effectiveness.

Experimental results

Research questions

• RQ1 Does increasing the number of retrieved passages consistently improve performance in long-context RAG systems?
• RQ2 To what extent do 'hard negatives' from strong retrievers degrade LLM generation quality in long-context RAG?
• RQ3 Can retrieval reordering mitigate attention inefficiencies in long-context LLMs?
• RQ4 How effective is implicit fine-tuning in improving robustness to noisy retrieved content?
• RQ5 Does adding an intermediate reasoning step during fine-tuning significantly improve relevance identification and final output quality?

Key findings

• Performance on long-context RAG tasks initially improves with more retrieved passages but eventually declines, peaking at 30 passages before dropping, indicating a trade-off between recall and noise.
• Stronger retrievers can introduce more harmful 'hard negatives,' worsening performance degradation compared to weaker retrievers.
• Retrieval reordering improves performance by up to 4.2% on average by reducing attention to middle-ranked, irrelevant passages.
• Explicit relevance fine-tuning with intermediate reasoning achieves the highest gains, improving accuracy by up to 12.7% over base models on TriviaQA and HotpotQA.
• Scaling RAG-specific fine-tuning data from 5k to 200k samples consistently improves performance, with accuracy increasing from 0.5805 to 0.6176 on NQ.
• Combining RAG-specific fine-tuning data with general SFT data preserves general capabilities (e.g., MT-Bench score) while boosting RAG performance, with NQ accuracy rising from 0.5687 to 0.6033.