[Paper Review] Are Large Language Models Memorizing Bug Benchmarks?

Daniel Ramos, Cláudia Mamede+4 more|arXiv (Cornell University)|Nov 20, 2024

Topic Modeling4 citations

TL;DR

This paper investigates whether large language models (LLMs) memorize widely used software bug benchmarks like Defects4J, using metrics including negative log-likelihood (NLL) and 5-gram accuracy. It finds that older models like codegen-multi exhibit strong memorization signals—especially for Defects4J—while newer models such as LLaMa 3.1 show significantly reduced leakage, highlighting the need for benchmark selection and robust evaluation practices to avoid inflated performance claims.

ABSTRACT

Large Language Models (LLMs) have become integral to various software engineering tasks, including code generation, bug detection, and repair. To evaluate model performance in these domains, numerous bug benchmarks containing real-world bugs from software projects have been developed. However, a growing concern within the software engineering community is that these benchmarks may not reliably reflect true LLM performance due to the risk of data leakage. Despite this concern, limited research has been conducted to quantify the impact of potential leakage. In this paper, we systematically evaluate popular LLMs to assess their susceptibility to data leakage from widely used bug benchmarks. To identify potential leakage, we use multiple metrics, including a study of benchmark membership within commonly used training datasets, as well as analyses of negative log-likelihood and n-gram accuracy. Our findings show that certain models, in particular codegen-multi, exhibit significant evidence of memorization in widely used benchmarks like Defects4J, while newer models trained on larger datasets like LLaMa 3.1 exhibit limited signs of leakage. These results highlight the need for careful benchmark selection and the adoption of robust metrics to adequately assess models capabilities.

Motivation & Objective

• To assess whether large language models (LLMs) are memorizing widely used software bug benchmarks, such as Defects4J and BugsInPy, due to data leakage from training data.
• To quantify the extent of memorization in LLMs using objective metrics like negative log-likelihood (NLL) and 5-gram accuracy.
• To compare leakage risks across different models, including older models (e.g., codegen-multi) and newer models (e.g., LLaMa 3.1) trained on larger datasets.
• To evaluate the reliability of established benchmarks by comparing performance on known benchmarks versus newly mined, likely unseen 2024 GitHub repositories.
• To raise awareness in the software engineering community about the risks of using benchmarks with potential memorization, which may inflate model performance metrics.

Proposed method

• The study uses negative log-likelihood (NLL) to measure model familiarity with code snippets from bug benchmarks and new repositories.
• It applies 5-gram accuracy to assess the model’s ability to reproduce exact sequences from benchmark solutions.
• The researchers analyze benchmark membership in TheStack, a widely used pretraining code dataset, to assess potential data leakage sources.
• They compare model performance on established benchmarks (e.g., Defects4J, BugsInPy) with performance on a newly collected dataset of high-quality, 2024-era GitHub repositories to isolate memorization from generalization.
• The analysis includes multiple LLMs, including codegen-multi, CodeLLaMa, LLaMa 3.1, Gemma 2, and others, across both code generation and repair tasks.
• The study uses trend analysis of NLL and 5-gram accuracy across models and datasets rather than relying on absolute values to mitigate noise from similar train/test splits.

Experimental results

Research questions

• RQ1 To what extent are popular LLMs, particularly codegen-multi, memorizing solutions from widely used bug benchmarks like Defects4J?
• RQ2 How do newer models like LLaMa 3.1, trained on larger and more diverse datasets, compare in terms of memorization risk on established benchmarks?
• RQ3 Do newer, less-established benchmarks such as GitBug-Java and BugsCpp show lower leakage signals compared to older benchmarks like Defects4J?
• RQ4 How do metrics like negative log-likelihood (NLL) and 5-gram accuracy detect data leakage in LLMs when applied to benchmark vs. novel code repositories?
• RQ5 What is the impact of model size and training data scale on the likelihood of memorizing benchmark solutions?

Key findings

• The model codegen-multi (6B parameters) exhibits strong memorization of Defects4J bug #39, reproducing the full solution verbatim, including comments and line breaks.
• Defects4J consistently shows the highest rate of potential memorization across all models and metrics, indicating it is the most vulnerable benchmark to data leakage.
• Older models like codegen-multi show very high 5-gram accuracy and low NLL on Defects4J, indicating significant memorization signals.
• Newer models such as LLaMa 3.1, trained on larger and more diverse datasets, display significantly reduced signs of memorization across all benchmarks.
• Newer benchmarks like GitBug-Java, BugsInPy, and BugsCpp show NLL and 5-gram accuracy levels comparable to those on newly mined 2024 GitHub repositories, suggesting lower leakage risk.
• The study confirms that data leakage is a real concern for benchmark evaluation, particularly when using older, widely used datasets like Defects4J, which may have been directly or indirectly included in training data.