[Paper Review] BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models

Zican Dong, Tianyi Tang+3 more|arXiv (Cornell University)|Sep 23, 2023

Topic ModelingComputer Science3 citations

TL;DR

BAMBOO is a comprehensive, multi-task benchmark designed to evaluate long text modeling capabilities of large language models (LLMs) across five diverse tasks—question answering, hallucination detection, text sorting, language modeling, and code completion—using two length levels (4k and 16k tokens). It addresses data contamination and evaluation accuracy issues, revealing that current LLMs struggle with long-context reasoning, instruction following, and unfamiliar tasks, with performance drops in longer inputs and limited generalization beyond fine-tuned domains.

ABSTRACT

Large language models (LLMs) have achieved dramatic proficiency over NLP tasks with normal length. Recently, multiple studies have committed to extending the context length and enhancing the long text modeling capabilities of LLMs. To comprehensively evaluate the long context ability of LLMs, we propose BAMBOO, a multi-task long context benchmark. BAMBOO has been designed with four principles: comprehensive capacity evaluation, avoidance of data contamination, accurate automatic evaluation, and different length levels. It consists of 10 datasets from 5 different long text understanding tasks, i.e. question answering, hallucination detection, text sorting, language modeling, and code completion, to cover core capacities and various domains of LLMs. We conduct experiments with five long context models on BAMBOO and further discuss four key research questions of long text. We also qualitatively analyze current long context models and point out future directions for enhancing long text modeling capacities. We release our data, prompts, and code at https://github.com/RUCAIBox/BAMBOO.

Motivation & Objective

• To address the lack of reliable, contamination-free, and automatically evaluable benchmarks for evaluating long-context LLMs.
• To comprehensively assess LLMs' abilities in long-range dependency modeling, knowledge utilization, reasoning, and tool use across diverse domains.
• To identify key challenges in long-context modeling, including instruction forgetting, format errors, and poor reasoning, especially in unfamiliar tasks.
• To provide a standardized, reproducible evaluation framework that supports future research in long-context LLM development.

Proposed method

• BAMBOO constructs 10 datasets from 5 core tasks—question answering, hallucination detection, text sorting, language modeling, and code completion—each designed to test distinct long-context capabilities.
• Datasets are built from 2023-released sources to prevent data contamination, ensuring evaluation integrity.
• The benchmark includes two length levels: BAMBOO-4k (avg. 2310 tokens) and BAMBOO-16k (avg. 6586 tokens), enabling analysis of length-dependent performance.
• Automatic evaluation metrics are carefully selected for accuracy and reproducibility, avoiding reliance on human annotation.
• Context compression techniques are applied to normal LLMs (e.g., ChatGPT, Vicuna) to compare performance against native long-context models.
• Qualitative analysis is conducted on instruction forgetting, format errors, reasoning failures, and performance on rare tasks.

Experimental results

Research questions

• RQ1 How do current long-context LLMs perform across diverse long-text tasks, and what are their key failure modes?
• RQ2 To what extent does input length affect model performance, and is there an 'extension tax' where long-context adaptation harms short-context performance?
• RQ3 How does the placement of task instructions within long inputs affect model adherence and performance?
• RQ4 Can context compression techniques effectively improve the performance of standard LLMs on long-context tasks?
• RQ5 Why do LLMs underperform on non-standard tasks like text sorting and code completion, even when provided with correct information?

Key findings

• ChatGPT-16k achieves the best overall performance across most BAMBOO datasets, while other models, including fine-tuned variants, show significant performance degradation on unfamiliar tasks.
• Performance generally declines with increasing input length, indicating an 'extension tax' where long-context adaptation impairs performance on shorter or simpler inputs.
• Instruction placement at the beginning of long inputs leads to catastrophic forgetting, especially in smaller models, due to insufficient instruction retention.
• Context compression techniques improve performance on some tasks (e.g., MeetingPred) but often degrade results on others (e.g., SenHallu), suggesting trade-offs in information preservation.
• Models exhibit poor reasoning abilities even when provided with correct evidence, indicating that reasoning limitations—not context length—are a primary bottleneck.
• Open-source models underperform on non-conversational, non-question-answering tasks like text sorting and code completion, due to narrow fine-tuning data distributions.