[论文解读] An Empirical Study of AI Generated Text Detection Tools

Arslan Akram|arXiv (Cornell University)|Sep 27, 2023

Artificial Intelligence in Healthcare and Education参考文献 24被引用 19

一句话总结

本论文在多域数据集上评估六种 AI 生成文本检测器，以评估它们对 ChatGPT 产出材料的有效性。

ABSTRACT

Since ChatGPT has emerged as a major AIGC model, providing high-quality responses across a wide range of applications (including software development and maintenance), it has attracted much interest from many individuals. ChatGPT has great promise, but there are serious problems that might arise from its misuse, especially in the realms of education and public safety. Several AIGC detectors are available, and they have all been tested on genuine text. However, more study is needed to see how effective they are for multi-domain ChatGPT material. This study aims to fill this need by creating a multi-domain dataset for testing the state-of-the-art APIs and tools for detecting artificially generated information used by universities and other research institutions. A large dataset consisting of articles, abstracts, stories, news, and product reviews was created for this study. The second step is to use the newly created dataset to put six tools through their paces. Six different artificial intelligence (AI) text identification systems, including "GPTkit," "GPTZero," "Originality," "Sapling," "Writer," and "Zylalab," have accuracy rates between 55.29 and 97.0%. Although all the tools fared well in the evaluations, originality was particularly effective across the board.

研究动机与目标

• 动机：需要在多域 ChatGPT 材料上评估检测器，而不仅仅是对真实文本进行评估。
• 创建一个大型、多域的数据集（文章、摘要、故事、新闻、产品评测）用于检测器评估。
• 在新数据集上评估前沿的检测工具，以衡量跨域性能。

提出的方法

• 组装一个由文章、摘要、故事、新闻和产品评测组成的大型多域数据集。
• 测试六种 AI 文本检测工具：GPTkit、GPTZero、Originality、Sapling、Writer 和 Zylalab。
• 在数据集上衡量每个工具的准确性，报告各工具的性能范围。

实验结果

研究问题

• RQ1 在来自多个领域的多样化、非真实内容上评估时，当前 AI 生成文本检测器的有效性如何？
• RQ2 领先检测工具的跨域检测准确性及变异性如何？
• RQ3 哪些检测器在跨领域表现最强且具有一致性？

主要发现

• 检测器的准确性在各工具之间从 55.29% 到 97.0% 之间变化。
• Originality 在所评估的工具中表现尤为出色。
• 检测器的性能因领域和工具而异。