[Paper Review] ChatGPT or Human? Detect and Explain. Explaining Decisions of Machine Learning Model for Detecting Short ChatGPT-generated Text

Sandra Mitrović, Davide Andreoletti+1 more|arXiv (Cornell University)|Jan 30, 2023

Topic Modeling81 citations

TL;DR

The paper trains a Transformer-based classifier to distinguish short ChatGPT-generated reviews from human-written ones and uses SHAP to explain model decisions, showing ML beats perplexity-based methods, especially when queries drive generation.

ABSTRACT

ChatGPT has the ability to generate grammatically flawless and seemingly-human replies to different types of questions from various domains. The number of its users and of its applications is growing at an unprecedented rate. Unfortunately, use and abuse come hand in hand. In this paper, we study whether a machine learning model can be effectively trained to accurately distinguish between original human and seemingly human (that is, ChatGPT-generated) text, especially when this text is short. Furthermore, we employ an explainable artificial intelligence framework to gain insight into the reasoning behind the model trained to differentiate between ChatGPT-generated and human-generated text. The goal is to analyze model's decisions and determine if any specific patterns or characteristics can be identified. Our study focuses on short online reviews, conducting two experiments comparing human-generated and ChatGPT-generated text. The first experiment involves ChatGPT text generated from custom queries, while the second experiment involves text generated by rephrasing original human-generated reviews. We fine-tune a Transformer-based model and use it to make predictions, which are then explained using SHAP. We compare our model with a perplexity score-based approach and find that disambiguation between human and ChatGPT-generated reviews is more challenging for the ML model when using rephrased text. However, our proposed approach still achieves an accuracy of 79%. Using explainability, we observe that ChatGPT's writing is polite, without specific details, using fancy and atypical vocabulary, impersonal, and typically it does not express feelings.

Motivation & Objective

• Assess whether a machine learning model can accurately distinguish short ChatGPT-generated text from human text in restaurant reviews.
• Evaluate the effectiveness of a perplexity-based baseline for short, AI-generated text detection.
• Provide explainable insights into the model's decisions to identify writing patterns of ChatGPT vs humans.
• Explore how rephrasing human reviews affects detection and model explanations.

Proposed method

• Fine-tune a DistilBERT-based transformer classifier on a dataset of human and ChatGPT-generated reviews.
• Compare ML-based classification with a perplexity-based approach using GPT-2 to compute perplexity scores and a threshold for labeling.
• Apply SHAP to obtain local explanations and identify features (words) driving each prediction.
• Use two experiments: (1) ChatGPT-generated reviews from custom queries vs human reviews, (2) ChatGPT-generated reviews rephrased from human ones.
• Analyze misclassifications to understand limitations when text is a rephrase of human content.

Figure 1: Schematic representation of the study design and building blocks.

Experimental results

Research questions

• RQ1 RQ1: How well do ML-based and perplexity-based approaches detect AI-generated text in short reviews, and how does performance change when AI text is a rephrase of human text?
• RQ2 RQ2: What textual features drive the ML model's decisions, and what do SHAP explanations reveal about ChatGPT vs human writing styles?

Key findings

• ML-based classifier achieves 0.98 accuracy in Experiment 1 and 0.79 in Experiment 2, outperforming the perplexity-based approach which scores 0.84 and 0.69 respectively.
• Perplexity scores alone can distinguish ChatGPT-generated vs human text in Experiment 1 but fail to reliably detect rephrased human text in Experiment 2.
• SHAP explanations show ChatGPT text is generally more polite, impersonal, and uses unusual vocabulary, while human text uses personal pronouns and expresses feelings, with more detailed specifics.
• Rephrasing human reviews reduces detection reliability, highlighting potential for AI-generated content to evade simple detection methods.
• Misclassifications in Experiment 2 reveal that local cues (e.g., pronouns like 'My', mention of specific items) can still lead to human-label predictions in ChatGPT-generated rephrases.