[论文解读] Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs

Simone Balloccu, Patrícia Schmidtová+2 more|arXiv (Cornell University)|Feb 6, 2024

Digital Rights Management and Security被引用 16

一句话总结

本研究系统性地分析 OpenAI GPT-3.5 与 GPT-4 的数据污染和评估不当，通过审阅 255 篇论文，发现约 42% 的信息泄漏，总计约 470 万样本，跨越 263 个基准，并凸显可复现性与公正性问题。

ABSTRACT

Natural Language Processing (NLP) research is increasingly focusing on the use of Large Language Models (LLMs), with some of the most popular ones being either fully or partially closed-source. The lack of access to model details, especially regarding training data, has repeatedly raised concerns about data contamination among researchers. Several attempts have been made to address this issue, but they are limited to anecdotal evidence and trial and error. Additionally, they overlook the problem of \emph{indirect} data leaking, where models are iteratively improved by using data coming from users. In this work, we conduct the first systematic analysis of work using OpenAI's GPT-3.5 and GPT-4, the most prominently used LLMs today, in the context of data contamination. By analysing 255 papers and considering OpenAI's data usage policy, we extensively document the amount of data leaked to these models during the first year after the model's release. We report that these models have been globally exposed to $\sim$4.7M samples from 263 benchmarks. At the same time, we document a number of evaluation malpractices emerging in the reviewed papers, such as unfair or missing baseline comparisons and reproducibility issues. We release our results as a collaborative project on https://leak-llm.github.io/, where other researchers can contribute to our efforts.

研究动机与目标

• 量化通过已发表研究对 GPT-3.5 和 GPT-4 的间接数据污染（数据泄漏）。
• 评估泄漏数据如何用于进一步训练，以及对公平比较与评估的潜在影响。
• 识别使用闭源大语言模型的研究中普遍存在的评估不当行为和可重复性障碍。
• 提出提升闭源 LLM 评估严谨性的最佳实践。

提出的方法

• 对评估 GPT-3.5 和 GPT-4 的 255 篇论文进行系统性文献综述。
• 识别哪些论文使用网页界面与 API 访问，并将潜在数据泄漏映射到 OpenAI 的数据使用政策。
• 在报告时按数据集和拆分估算泄漏数据；若未报告，则假设使用整个数据集。
• 评估提示可用性、代码仓库、数据集拆分以及模型版本报告，以提高可重复性。
• 通过与基线进行比较并确保跨模型使用数据的一致性，分析评估公平性。

Figure 1: Distribution of the dates when papers evaluating ChatGPT were first uploaded to arXiv or published. The dotted line represents the ChatGPT API release (March 1st, 2023, dotted line in the chart) as a cutoff point. The single paper shown using the API in February is by a research group that

实验结果

研究问题

• RQ1 在过去一年中，哪些数据集被明确泄漏给 GPT-3.5 与 GPT-4？
• RQ2 评估这些模型的论文是否包含与现有基线的公平比较？
• RQ3 在闭源 LLM 评估中，哪些做法普遍削弱可重复性和公平性？
• RQ4 哪些指南可以减轻闭源 LLM 的数据污染和评估不当行为？

主要发现

• ~42% 的已审阅论文将数据泄漏给 GPT-3.5 或 GPT-4，总计 ~4.7 million 样本，跨越 263 个基准。
• 90 篇论文（~42%）通过网页界面访问 ChatGPT，暴露了 OpenAI 可用于训练的数据。
• 泄漏的数据集分布在各类任务中，在自然语言推理、问答和自然语言生成方面的泄漏量较高。
• 仅 ~91% 报告了使用的提示；~53% 提供了代码仓库；仅 40%/23% 提供了模型版本细节，分别对应同行评审论文/预印本论文。
• 许多研究进行了不公平或不完整的比较，通常在样本量上对 ChatGPT 的评估少于开源模型，且数据规模不同。
• 一个协作性的公共资源（leak-llm.github.io）被发布，用于记录泄漏并鼓励社区贡献。

Figure 2: Data leakage distribution. We report the number of times (y) we observed a specific percentage of leaking (x) for the considered split. As some work vaguely describes the used split as “test or dev set”, we merge these two values in a unique chart.