[论文解读] Assessing AI Detectors in Identifying AI-Generated Code: Implications for Education

Wei Pan, Ming Jie Chok+7 more|arXiv (Cornell University)|Jan 8, 2024

Artificial Intelligence in Healthcare and Education被引用 6

一句话总结

该论文在13种提示变体下，对AI生成内容（AIGC）检测器对AI生成的Python代码进行了5,069样本的数据集评估，并发现检测器在区分人类编写与AI生成的代码方面表现较差，对代码变体也很敏感。

ABSTRACT

Educators are increasingly concerned about the usage of Large Language Models (LLMs) such as ChatGPT in programming education, particularly regarding the potential exploitation of imperfections in Artificial Intelligence Generated Content (AIGC) Detectors for academic misconduct. In this paper, we present an empirical study where the LLM is examined for its attempts to bypass detection by AIGC Detectors. This is achieved by generating code in response to a given question using different variants. We collected a dataset comprising 5,069 samples, with each sample consisting of a textual description of a coding problem and its corresponding human-written Python solution codes. These samples were obtained from various sources, including 80 from Quescol, 3,264 from Kaggle, and 1,725 from LeetCode. From the dataset, we created 13 sets of code problem variant prompts, which were used to instruct ChatGPT to generate the outputs. Subsequently, we assessed the performance of five AIGC detectors. Our results demonstrate that existing AIGC Detectors perform poorly in distinguishing between human-written code and AI-generated code.

研究动机与目标

• 评估现有AIGC检测器在编程任务中识别AI生成代码的有效性。
• 在多样化的代码和提示变体下调查检测器的局限性。
• 提供实证数据以为教育诚信实践与检测器使用提供依据。

提出的方法

• 汇集一个包含5,069个样本的数据集，样本包括来自Quescol、Kaggle和LeetCode的题目描述和人类编写的Python解答。
• 通过变体提示对ChatGPT进行提示，创建13个AI生成代码变体。
• 使用0.5的判定阈值，应用五个AIGC检测器将代码分类为人类或AI生成。
• 在所有变体中使用准确度、精确度、TPR、FPR、TNR和FNR来评估检测器。
• 提供一个复现包并以算法步骤记录工作流程以确保可重复性。

实验结果

研究问题

• RQ1 RQ1：现有AIGC检测器在检测AI生成代码方面的准确性如何？
• RQ2 RQ2：在检测AI生成代码方面，现有AIGC检测器的局限性是什么？

主要发现

• 所有五个检测器在区分人类编写与AI生成代码方面显示出较差的准确性（若干检测器的ACC约为0.5）。
• GLTR对变体的敏感性显著，ACC在0.4841到0.7693之间变化。
• DetectGPT在相对于某些基线的基础上将真阴性率（TNR）提升了5–9个百分点，但总体仍然有限。
• Sapling在许多变体中始终实现ACC超过0.6，显示对某些变体的相对鲁棒性。
• GPTZero和GPT-2 Detector在多种变体中通常徘徊在基线性能附近，具有较低的真阴性率。
• 变体引发的变化（例如去除停用词、命名更改、死代码）可能显著影响检测器性能。