[Paper Review] Long Code Arena: a Set of Benchmarks for Long-Context Code Models

Egor Bogomolov, Aleksandra Eliseeva+9 more|arXiv (Cornell University)|Jun 17, 2024

Distributed and Parallel Computing Systems4 citations

TL;DR

This paper introduces Long Code Arena, a benchmark suite of six tasks requiring long-context code understanding across entire projects, including library-based generation, CI repair, project-level completion, commit message creation, bug localization, and module summarization. Built from permissively licensed open-source repositories with rigorous manual validation, it provides high-quality datasets, evaluation tools, and baselines on HuggingFace and GitHub to advance long-context code model research.

ABSTRACT

Nowadays, the fields of code and natural language processing are evolving rapidly. In particular, models become better at processing long context windows - supported context sizes have increased by orders of magnitude over the last few years. However, there is a shortage of benchmarks for code processing that go beyond a single file of context, while the most popular ones are limited to a single method. With this work, we aim to close this gap by introducing Long Code Arena, a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions based on popular LLMs to showcase the usage of the dataset and to simplify adoption by other researchers. We publish the benchmark page on HuggingFace Spaces with the leaderboard, links to HuggingFace Hub for all the datasets, and link to the GitHub repository with baselines: https://huggingface.co/spaces/JetBrains-Research/long-code-arena.

Motivation & Objective

• To address the lack of benchmarks that evaluate code models on project-wide context beyond single files or methods.
• To create realistic, high-fidelity evaluation tasks that reflect real-world software engineering workflows requiring access to multiple files or an entire module.
• To ensure data quality through manual verification and filtering of samples from real open-source repositories with permissive licenses.
• To provide open-source evaluation tools, datasets on HuggingFace Hub, and baseline implementations to accelerate research in long-context code models.
• To enable fair and reproducible evaluation of long-context LLMs on complex, project-scale code processing tasks.

Proposed method

• The benchmark is constructed from 4,343 open-source GitHub repositories filtered for size, activity, and permissive licensing to ensure data quality and reusability.
• For each of the six tasks, data is extracted from source code, commit history, issues, and GitHub Actions logs to simulate real-world development contexts.
• Rigorous filtering and manual validation are applied to ensure correctness and relevance, especially for tasks like commit message generation and bug localization.
• A deep learning classifier (fine-tuned CodeBERT) is used to filter commit messages based on quality criteria such as capitalization and verb usage.
• Datasets are published on HuggingFace Hub with a public leaderboard, and baseline models are implemented using popular LLMs to demonstrate usage.
• The evaluation setup ensures disjoint training and test sets, with a focus on maintaining temporal consistency to avoid data leakage.

Experimental results

Research questions

• RQ1 How well can long-context code models perform on tasks requiring understanding of an entire project or module, rather than isolated functions?
• RQ2 To what extent do existing benchmarks fail to reflect the complexity of real-world software engineering workflows?
• RQ3 Can a curated, manually verified dataset of project-scale code tasks improve the reliability and reproducibility of model evaluations?
• RQ4 How do popular LLMs perform on long-context code tasks when evaluated on a realistic, project-wide benchmark?
• RQ5 What are the key challenges in building and maintaining high-quality, scalable benchmarks for long-context code models?

Key findings

• The Long Code Arena benchmark includes six distinct tasks requiring project-wide context, such as CI build repair and module summarization, which are absent in most existing benchmarks.
• The dataset contains 100 to 1,000 manually verified examples per task, extracted from 4,343 high-quality open-source repositories with permissive licenses.
• A deep learning classifier based on CodeBERT was fine-tuned to filter commit messages, improving the quality of the commit message generation dataset.
• The benchmark is hosted on HuggingFace Spaces with a public leaderboard and all datasets are available on HuggingFace Hub for community use.
• Baseline implementations are provided on GitHub, demonstrating the use of popular LLMs on each task to serve as a reference point for future research.
• The dataset construction process avoids data leakage by ensuring temporal consistency and using disjoint training and test sets.