[Paper Review] NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities

Mo Li, Songyang Zhang+4 more|arXiv (Cornell University)|Jul 16, 2024

Natural Language Processing Techniques4 citations

TL;DR

NeedleBench introduces a comprehensive benchmark for evaluating long-context LLMs across multiple lengths (4k to 1M+ tokens) and depths, focusing on retrieval and reasoning in bilingual (English/Chinese) contexts. It proposes the Ancestral Trace Challenge (ATC) to test multi-step logical reasoning, revealing that even state-of-the-art models struggle significantly with complex reasoning in long contexts, despite strong performance on basic retrieval tasks.

ABSTRACT

The capability of large language models to handle long-context information is crucial across various real-world applications. Existing evaluation methods often rely either on real-world long texts, making it difficult to exclude the influence of models' inherent knowledge, or introduce irrelevant filler content to artificially achieve target lengths, reducing assessment effectiveness. To address these limitations, we introduce NeedleBench, a synthetic framework for assessing retrieval and reasoning performance in bilingual long-context tasks with adaptive context lengths. NeedleBench systematically embeds key data points at varying depths to rigorously test model capabilities. Tasks are categorized into two scenarios: information-sparse, featuring minimal relevant details within extensive irrelevant text to simulate simple retrieval tasks; and information-dense (the Ancestral Trace Challenge), where relevant information is continuously distributed throughout the context to simulate complex reasoning tasks. Our experiments reveal that although recent reasoning models like Deepseek-R1 and OpenAI's o3 excel in mathematical reasoning, they struggle with continuous retrieval and reasoning in information-dense scenarios, even at shorter context lengths. We also characterize a phenomenon termed 'under-thinking', where models prematurely conclude reasoning despite available information. NeedleBench thus provides critical insights and targeted tools essential for evaluating and improving LLMs' long-context capabilities. All resources are available at OpenCompass: https://github.com/open-compass/opencompass.

Motivation & Objective

• To address the misalignment between existing long-context evaluation benchmarks and real-world tasks requiring multi-step reasoning and information integration.
• To develop a scalable, customizable benchmark framework that supports progressive evaluation across diverse context lengths and depth zones.
• To introduce the Ancestral Trace Challenge (ATC) as a proxy for real-world complex reasoning tasks in long-context scenarios.
• To assess the performance of leading open-source LLMs in bilingual (English/Chinese) long-context understanding, especially under high-complexity reasoning conditions.
• To identify critical limitations in current models’ instruction-following, prompt sensitivity, and reasoning robustness in long-context settings.

Proposed method

• Design a multi-length, multi-depth benchmark framework with strategically inserted 'needles' (key information points) across 4k, 8k, 32k, 128k, 200k, 1M, and beyond token contexts.
• Implement the Ancestral Trace Challenge (ATC) as a simplified proxy for complex long-context reasoning, requiring models to trace logical dependencies across multiple dispersed facts.
• Conduct controlled experiments varying prompt position (before or after context) to assess model sensitivity and instruction-following behavior.
• Evaluate retrieval performance in multi-needle tasks with increasing numbers of required key information points to test reasoning and recall robustness.
• Use bilingual datasets (English and Chinese) to ensure cross-lingual evaluation of long-context capabilities.
• Analyze model behavior across different context lengths and prompt placements to identify performance degradation patterns and sensitivity issues.

Figure 1: Our ATC test, even with a modest context length of 2K, reveals surprising limitations of these leading models in real-world complex long-context questions.

Experimental results

Research questions

• RQ1 Can current LLMs reliably retrieve a single key fact from a 1M-token context, even when it is deeply embedded?
• RQ2 How does the number of required retrieval points affect model performance in multi-needle reasoning tasks?
• RQ3 Does placing the question prompt before or after the long context significantly alter model performance?
• RQ4 To what extent do models generalize reasoning capabilities beyond simple retrieval, especially in complex, multi-step logical scenarios?
• RQ5 How sensitive are long-context LLMs to prompt position and instruction phrasing in real-world long-context applications?

Key findings

• All leading open-source LLMs show significant performance degradation on the Ancestral Trace Challenge (ATC), even at context lengths below 2K tokens, indicating fundamental weaknesses in multi-step reasoning.
• Models exhibit paradoxical improvements in initial needle recall when asked to retrieve more needles, suggesting poor instruction-following robustness and inconsistent utilization of retrieval capabilities.
• Prompt position has a substantial impact: placing the question before the context generally degrades performance, though some models like InternLM2-7B-200K benefit from early prompting in specific tasks.
• The performance of models on retrieval tasks does not generalize to reasoning tasks, highlighting a critical gap between isolated fact-finding and integrated logical inference.
• Current models are highly sensitive to prompt placement and phrasing, indicating a lack of robustness in long-context reasoning under real-world conditions.
• Despite strong performance on basic passkey-style retrieval, models fail on complex, multi-fact reasoning tasks, revealing that long-context capability is not yet sufficient for practical, real-world applications.

Figure 2: NeedleBench Framework