[Paper Review] Large Language Model (LLM) AI text generation detection based on transformer deep learning algorithm

Yuhong Mo, Hao Qin+3 more|arXiv (Cornell University)|Apr 6, 2024

Computational and Text Analysis Methods23 citations

TL;DR

The paper develops a transformer-based detection tool for LLM-generated text, achieving very high accuracy on validation and test sets.

ABSTRACT

In this paper, a tool for detecting LLM AI text generation is developed based on the Transformer model, aiming to improve the accuracy of AI text generation detection and provide reference for subsequent research. Firstly the text is Unicode normalised, converted to lowercase form, characters other than non-alphabetic characters and punctuation marks are removed by regular expressions, spaces are added around punctuation marks, first and last spaces are removed, consecutive ellipses are replaced with single spaces and the text is connected using the specified delimiter. Next remove non-alphabetic characters and extra whitespace characters, replace multiple consecutive whitespace characters with a single space and again convert to lowercase form. The deep learning model combines layers such as LSTM, Transformer and CNN for text classification or sequence labelling tasks. The training and validation sets show that the model loss decreases from 0.127 to 0.005 and accuracy increases from 94.96 to 99.8, indicating that the model has good detection and classification ability for AI generated text. The test set confusion matrix and accuracy show that the model has 99% prediction accuracy for AI-generated text, with a precision of 0.99, a recall of 1, and an f1 score of 0.99, achieving a very high classification accuracy. Looking forward, it has the prospect of wide application in the field of AI text detection.

Motivation & Objective

• Motivate the need for accurate detection of AI-generated text in various applications.
• Develop a detection model that leverages transformer-based deep learning for classification.
• Evaluate the model on validation and test sets to demonstrate detection performance.

Proposed method

• Preprocess text with Unicode normalization and regex-based cleaning, followed by lowercasing and punctuation handling.
• Combine LSTM, Transformer, and CNN layers for text classification or sequence labeling tasks.
• Train and validate the model, reporting loss and accuracy improvements across stages (loss 0.127 to 0.005; accuracy 94.96 to 99.8).
• Evaluate on a test set with a confusion matrix, reporting accuracy, precision, recall, and F1-score.

Experimental results

Research questions

• RQ1 Can a transformer-based deep learning model effectively detect AI-generated text across different datasets?
• RQ2 What preprocessing steps maximize the detector's performance for LLM-generated content?
• RQ3 How do combined architectures (LSTM + Transformer + CNN) compare to single-architecture detectors in terms of accuracy and reliability?

Key findings

• The model achieved validation accuracy increasing from 94.96 to 99.8 and a corresponding loss decrease from 0.127 to 0.005.
• On the test set, the detector achieved 99% prediction accuracy for AI-generated text.
• Precision on the test set was 0.99, recall was 1, and F1-score was 0.99.
• The results indicate very high classification accuracy and strong detection capability for AI-generated text.