[Paper Review] Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation

Yucheng Li|arXiv (Cornell University)|Sep 19, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a perplexity-based method to estimate data contamination in language model benchmarks without access to training data. By comparing test set perplexity against memorised and clean baselines from the same source distribution, the method detects significant memorisation in reading comprehension and summarisation benchmarks—especially in larger models—while finding minimal contamination in multiple-choice datasets like MMLU.

ABSTRACT

Data contamination in model evaluation is getting increasingly prevalent as the massive training corpora of large language models often unintentionally include benchmark samples. Therefore, contamination analysis has became an inevitable part of reliable model evaluation. However, existing method of contamination analysis requires the access of the entire training data which is often confidential for recent models. This prevent the community to rigorously audit these models and conduct accurate assessment of their capability. In this paper, we propose a novel method to quantify contamination without the access of the full training set, that measure the extent of contamination with perplexity. Our analysis provides evidence of significant memorisation of recent foundation models in popular reading comprehension, summarisation benchmarks, while multiple choice appears less contaminated.

Motivation & Objective

• To address the lack of reliable contamination analysis for closed-source and fine-tuned language models due to unavailable training data.
• To develop a method that quantifies memorisation in evaluation benchmarks without requiring full access to training corpora.
• To enable community-wide auditing of model capabilities by providing a scalable, accessible alternative to n-gram overlap analysis.
• To evaluate the extent of data contamination across major NLP benchmarks using a proxy measure based on model likelihood.

Proposed method

• The method compares the perplexity of a test set to two baselines: one derived from training data (memorised baseline) and one from data not in the training set (clean baseline).
• Baselines are constructed from the same source distribution as the test set—e.g., Wikipedia for reading comprehension, BBC news for summarisation—ensuring comparable format and length.
• For each benchmark, the model's perplexity on the test set is measured and compared to the average perplexity of the memorised and clean baselines.
• A test set with perplexity closer to the memorised baseline indicates significant memorisation and contamination.
• The approach uses verbalisation techniques to format inputs consistently for perplexity computation, especially for multi-choice questions.
• The method avoids expensive n-gram overlap computation by directly measuring model likelihood via perplexity.

Experimental results

Research questions

• RQ1 To what extent are popular reading comprehension benchmarks contaminated in large language models when training data is inaccessible?
• RQ2 Can perplexity differences between test sets and baselines reliably indicate memorisation and contamination?
• RQ3 How does contamination vary across different types of benchmarks—reading comprehension, summarisation, and multiple-choice questions?
• RQ4 Are smaller models less prone to memorisation than larger ones in the absence of training data access?

Key findings

• All tested foundation models—gpt-3, llama-7b, 13b, and 30b—exhibit significant memorisation in reading comprehension benchmarks like QuAC, BoolQ, and SQuAD_v2, with perplexity values closer to the memorised baseline.
• The LLaMA-30B model shows particularly strong memorisation, achieving lower perplexity on SQuAD than even its own memorised baseline.
• Summarisation benchmarks such as XSum also show strong signs of contamination, with perplexity values near the memorised baseline across all models.
• In contrast, multiple-choice benchmarks like MMLU show no clear contamination trend, with perplexity values not correlated to time of data creation, suggesting minimal memorisation.
• Smaller models like LLaMA-7B show less memorisation than larger models, but still exhibit measurable contamination in reading comprehension tasks.
• The method successfully identifies contamination in benchmarks without requiring access to training data, enabling credible evaluation of closed-source and fine-tuned models.