[Paper Review] In Defense of RAG in the Era of Long-Context Language Models

Yu Tan, Anbang Xu+1 more|arXiv (Cornell University)|Sep 3, 2024

Topic Modeling4 citations

TL;DR

This paper proposes Order-Preserve Retrieval-Augmented Generation (OP-RAG), a method that maintains the original order of retrieved document chunks to improve answer quality in long-context question-answering. Unlike vanilla RAG or long-context LLMs, OP-RAG achieves higher F1 scores (up to 47.25) with significantly fewer input tokens (48K) than long-context models processing 117K–196K tokens, demonstrating that focused retrieval outperforms brute-force context length extension.

ABSTRACT

Overcoming the limited context limitations in early-generation LLMs, retrieval-augmented generation (RAG) has been a reliable solution for context-based answer generation in the past. Recently, the emergence of long-context LLMs allows the models to incorporate much longer text sequences, making RAG less attractive. Recent studies show that long-context LLMs significantly outperform RAG in long-context applications. Unlike the existing works favoring the long-context LLM over RAG, we argue that the extremely long context in LLMs suffers from a diminished focus on relevant information and leads to potential degradation in answer quality. This paper revisits the RAG in long-context answer generation. We propose an order-preserve retrieval-augmented generation (OP-RAG) mechanism, which significantly improves the performance of RAG for long-context question-answer applications. With OP-RAG, as the number of retrieved chunks increases, the answer quality initially rises, and then declines, forming an inverted U-shaped curve. There exist sweet points where OP-RAG could achieve higher answer quality with much less tokens than long-context LLM taking the whole context as input. Extensive experiments on public benchmark demonstrate the superiority of our OP-RAG.

Motivation & Objective

• To re-evaluate the role of RAG in the era of long-context LLMs, challenging the assumption that long-context models alone outperform retrieval-augmented methods.
• To investigate whether the order of retrieved chunks affects answer quality in long-context question-answering.
• To identify optimal retrieval scope where RAG performance peaks before degrading due to irrelevant information.
• To demonstrate that efficient, focused retrieval can surpass the performance of long-context LLMs processing entire documents.

Proposed method

• Propose Order-Preserve RAG (OP-RAG), which preserves the original sequence of retrieved document chunks instead of reordering them by relevance.
• Use dense vector retrieval (BGE-large-en-v1.5) to embed queries and document chunks, retrieving top-k chunks based on similarity.
• Structure the input context for the LLM by concatenating retrieved chunks in their original document order, maintaining temporal and structural coherence.
• Systematically vary the number of retrieved chunks to study the trade-off between recall (more relevant info) and precision (less distraction).
• Use Llama3.1-70B and other long-context LLMs as generators to evaluate performance across different input lengths.
• Apply ablation studies to isolate the impact of retrieval order and context length on answer quality.

(a) F1 score.

Experimental results

Research questions

• RQ1 Does preserving the original order of retrieved chunks improve answer quality in long-context question-answering compared to relevance-sorted retrieval?
• RQ2 How does answer quality in RAG vary with increasing numbers of retrieved chunks, and does it follow an inverted U-shaped curve?
• RQ3 Can RAG with optimized chunk count outperform long-context LLMs that process entire documents as input, despite using fewer tokens?
• RQ4 What is the optimal balance between retrieval recall and distraction in long-context RAG systems?
• RQ5 How does the model scale (e.g., Llama3.1-8B vs. 70B) affect the peak performance point in OP-RAG?

Key findings

• OP-RAG achieves a 47.25 F1 score on the En.QA benchmark using only 48K input tokens, outperforming Llama3.1-70B without RAG, which achieves 34.26 F1 with 117K tokens.
• With 128 retrieved chunks, OP-RAG achieves 44.43 F1 on En.QA, while vanilla RAG with the same number of chunks reaches only 38.40 F1.
• On the EN.MC dataset, OP-RAG with 192 chunks achieves 88.65% accuracy, compared to 81.22% for vanilla RAG, demonstrating a significant gain at scale.
• The performance of OP-RAG follows an inverted U-shaped curve: quality improves with more chunks up to a peak, then declines due to increased distraction from irrelevant content.
• OP-RAG with 24K tokens achieves 88.65% accuracy on EN.MC, surpassing GPT-4O (78.42%) and Gemini-1.5-Pro (85.57%) despite using far fewer tokens.
• The peak performance for Llama3.1-70B occurs at 48K tokens on En.QA and 32K on EN.MC, indicating that larger models can tolerate more context before degradation.

(b) Input token count.