[论文解读] Large Language Model (LLM) AI text generation detection based on transformer deep learning algorithm

Yuhong Mo, Hao Qin+3 more|arXiv (Cornell University)|Apr 6, 2024

Computational and Text Analysis Methods被引用 23

一句话总结

本论文开发了基于 transformer 的用于检测 LLM 生成文本的工具，在验证集和测试集上达到非常高的准确率。

ABSTRACT

In this paper, a tool for detecting LLM AI text generation is developed based on the Transformer model, aiming to improve the accuracy of AI text generation detection and provide reference for subsequent research. Firstly the text is Unicode normalised, converted to lowercase form, characters other than non-alphabetic characters and punctuation marks are removed by regular expressions, spaces are added around punctuation marks, first and last spaces are removed, consecutive ellipses are replaced with single spaces and the text is connected using the specified delimiter. Next remove non-alphabetic characters and extra whitespace characters, replace multiple consecutive whitespace characters with a single space and again convert to lowercase form. The deep learning model combines layers such as LSTM, Transformer and CNN for text classification or sequence labelling tasks. The training and validation sets show that the model loss decreases from 0.127 to 0.005 and accuracy increases from 94.96 to 99.8, indicating that the model has good detection and classification ability for AI generated text. The test set confusion matrix and accuracy show that the model has 99% prediction accuracy for AI-generated text, with a precision of 0.99, a recall of 1, and an f1 score of 0.99, achieving a very high classification accuracy. Looking forward, it has the prospect of wide application in the field of AI text detection.

研究动机与目标

• 在各种应用中证明需要准确检测 AI 生成文本的动机。
• 开发一个利用基于 transformer 的深度学习进行分类的检测模型。
• 在验证集和测试集上评估模型，以展示检测性能。

提出的方法

• 使用 Unicode 归一化和基于正则表达式的清理对文本进行预处理，随后进行小写化和标点处理。
• 将 LSTM、Transformer 和 CNN 层结合用于文本分类或序列标注任务。
• 训练并验证模型，报告各阶段的损失和准确性提升（loss 0.127 到 0.005；accuracy 94.96 到 99.8）。
• 在测试集上使用混淆矩阵进行评估，报告准确率、精确率、召回率和 F1-score。

实验结果

研究问题

• RQ1 一个基于 transformer 的深度学习模型是否能够在不同数据集上有效检测 AI 生成的文本？
• RQ2 哪些预处理步骤能最大化检测器对 LLM 生成内容的性能？
• RQ3 组合架构（LSTM + Transformer + CNN）在准确性和可靠性方面与单架构检测器相比有何差异？

主要发现

• 模型在验证集上的准确率从 94.96 提升到 99.8，损失从 0.127 降至 0.005。
• 在测试集上，检测器对 AI 生成文本的预测准确率达到 99%。
• 测试集的精确度为 0.99，召回率为 1，F1-score 为 0.99。
• 结果表明极高的分类准确性和对 AI 生成文本的强检测能力。