[Paper Review] Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems

Philippe Laban, Alexander R. Fabbri+2 more|arXiv (Cornell University)|Jul 1, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper introduces SummHay, a novel benchmark for evaluating long-context LLMs and RAG systems by synthesizing large document collections ('Haystacks') with repeated, controlled insights across documents. Systems must summarize query-relevant insights and cite source documents precisely. The evaluation reveals that even with oracle retrieval, models lag behind human performance by over 10 points on joint coverage and citation score, exposing significant limitations in current long-context reasoning and retrieval systems.

ABSTRACT

LLMs and RAG systems are now capable of handling millions of input tokens or more. However, evaluating the output quality of such systems on long-context tasks remains challenging, as tasks like Needle-in-a-Haystack lack complexity. In this work, we argue that summarization can play a central role in such evaluation. We design a procedure to synthesize Haystacks of documents, ensuring that specific extit{insights} repeat across documents. The "Summary of a Haystack" (SummHay) task then requires a system to process the Haystack and generate, given a query, a summary that identifies the relevant insights and precisely cites the source documents. Since we have precise knowledge of what insights should appear in a haystack summary and what documents should be cited, we implement a highly reproducible automatic evaluation that can score summaries on two aspects - Coverage and Citation. We generate Haystacks in two domains (conversation, news), and perform a large-scale evaluation of 10 LLMs and corresponding 50 RAG systems. Our findings indicate that SummHay is an open challenge for current systems, as even systems provided with an Oracle signal of document relevance lag our estimate of human performance (56\%) by 10+ points on a Joint Score. Without a retriever, long-context LLMs like GPT-4o and Claude 3 Opus score below 20% on SummHay. We show SummHay can also be used to study enterprise RAG systems and position bias in long-context models. We hope future systems can equal and surpass human performance on SummHay.

Motivation & Objective

• To address the lack of robust, reproducible evaluation for long-context LLMs and RAG systems in complex, multi-document summarization tasks.
• To overcome limitations of existing benchmarks like Needle-in-a-Haystack, which lack complexity and fail to differentiate state-of-the-art models.
• To develop a scalable, automatic evaluation protocol based on coverage of reference insights and citation accuracy, enabling reliable comparison across systems.
• To estimate human performance on long-context summarization and benchmark 10 LLMs and 50 RAG systems under realistic and oracle retrieval conditions.
• To identify key failure modes such as position bias and poor retrieval generalization in long-context models.

Proposed method

• Synthesizing Haystacks using a controlled data generation pipeline that ensures specific insights repeat across 100 documents in two domains: conversations and news.
• Designing the SummHay task to require models to generate summaries that cover relevant insights and cite the exact source documents, with precise control over expected outputs.
• Implementing a dual evaluation protocol: manual annotation with high inter-annotator agreement (0.77) and LLM-based evaluation with moderate correlation (0.71) and 50x lower cost.
• Generating 10 Haystacks, each with ~10 queries, totaling 92 SummHay tasks, with insights categorized by subtopics and source documents precisely tracked.
• Evaluating 10 closed-source LLMs and 50 RAG systems under both standard and oracle retrieval settings to isolate model and retrieval performance.
• Using a combination of automatic metrics and human-validated reference summaries to measure coverage and citation quality, with a joint score for overall performance.

Experimental results

Research questions

• RQ1 Can summarization serve as a robust, complex benchmark for evaluating long-context LLMs and RAG systems beyond simple retrieval tasks like Needle-in-a-Haystack?
• RQ2 How do current long-context LLMs and RAG systems perform on a multi-document summarization task requiring precise insight coverage and citation?
• RQ3 To what extent does the inclusion of an oracle retriever improve system performance on the SummHay benchmark?
• RQ4 How does human performance compare to model performance on the SummHay task, and can models surpass it?
• RQ5 What are the key failure modes in current systems, such as position bias or poor citation accuracy, and how do they affect performance?

Key findings

• Even with an oracle signal identifying relevant documents, all evaluated models scored below 56% on the joint coverage and citation score, lagging behind the estimated human performance by over 10 points.
• Long-context LLMs like GPT-4o and Claude 3 Opus scored below 20% on SummHay without a retriever, indicating significant challenges in reasoning over long contexts.
• The LLM-based evaluation method achieved a correlation of 0.71 with human judgments, offering a cost-effective alternative to manual annotation.
• The manual evaluation protocol showed strong reproducibility among annotators, with a 0.77 inter-annotator correlation, validating the reliability of the benchmark.
• Position bias and poor citation accuracy were identified as key failure modes, particularly in long-context models without retrieval.
• The benchmark revealed that current RAG systems and LLMs struggle with both coverage and citation precision, even under optimal retrieval conditions, highlighting a major open challenge in long-context AI.