[Paper Review] GPT-who: An Information Density-based Machine-Generated Text Detector

Saranya Venkatraman, Adaku Uchendu+1 more|arXiv (Cornell University)|Oct 9, 2023

Topic ModelingComputer Science3 citations

TL;DR

GPT-who is a psycholinguistically-informed, domain-agnostic, statistical detector that uses Uniform Information Density (UID)-based features to distinguish between human and multiple LLM-generated texts without fine-tuning. It outperforms state-of-the-art detectors by over 20% in multi-class authorship attribution across diverse domains and LLMs, while remaining computationally efficient and interpretable via surprisal-based feature analysis.

ABSTRACT

The Uniform Information Density (UID) principle posits that humans prefer to spread information evenly during language production. We examine if this UID principle can help capture differences between Large Language Models (LLMs)-generated and human-generated texts. We propose GPT-who, the first psycholinguistically-inspired domain-agnostic statistical detector. This detector employs UID-based features to model the unique statistical signature of each LLM and human author for accurate detection. We evaluate our method using 4 large-scale benchmark datasets and find that GPT-who outperforms state-of-the-art detectors (both statistical- & non-statistical) such as GLTR, GPTZero, DetectGPT, OpenAI detector, and ZeroGPT by over $20$% across domains. In addition to better performance, it is computationally inexpensive and utilizes an interpretable representation of text articles. We find that GPT-who can distinguish texts generated by very sophisticated LLMs, even when the overlying text is indiscernible. UID-based measures for all datasets and code are available at https://github.com/saranya-venkatraman/gpt-who.

Motivation & Objective

• To develop a robust, multi-class, domain-agnostic detector for identifying the authorship of machine-generated text across diverse LLMs and writing tasks.
• To leverage the Uniform Information Density (UID) principle as a theoretically grounded, interpretable feature space for detecting subtle statistical differences between human and LLM-generated language.
• To create a computationally efficient detector that avoids expensive fine-tuning while maintaining high performance across out-of-distribution and unseen model settings.
• To demonstrate that UID-based features can reliably distinguish between different LLM families and human authors, even when textual differences are imperceptible.
• To provide an interpretable alternative to black-box LLM-based detectors by exposing the surprisal patterns underlying authorship decisions.

Proposed method

• The method computes token-level surprisal probabilities using an off-the-shelf language model to extract UID-based features reflecting the distribution of information density in text.
• It constructs a set of theoretically motivated, interpretable features derived from the statistical distribution of token probabilities, capturing how evenly information is spread across sentences and phrases.
• A logistic regression classifier is trained on these UID features to learn thresholds that distinguish between human authors and multiple LLMs.
• The detector is trained and evaluated on four large-scale benchmark datasets: TuringBench, GPABenchmark, ArguGPT, and Deepfake Text in-the-wild.
• The approach is domain-agnostic and task-agnostic, enabling generalization across writing styles, topics, and LLM architectures without retraining or fine-tuning.
• Feature interpretability is enhanced by visualizing and analyzing surprisal patterns, enabling insight into model decision-making.

Figure 1: GPT-who leverages psycholinguistically motivated representations that capture authors’ information signatures distinctly, even when the corresponding text is indiscernible.

Experimental results

Research questions

• RQ1 Can the Uniform Information Density (UID) principle serve as a reliable statistical signature to differentiate between human and LLM-generated text?
• RQ2 Can UID-based features enable accurate multi-class authorship attribution across diverse LLMs and domains without model fine-tuning?
• RQ3 How do the UID distributions of human-generated texts differ from those of various LLMs, especially in terms of information spread and architectural similarity?
• RQ4 To what extent can UID-based features outperform both statistical and deep learning-based detectors in zero-shot and out-of-distribution settings?
• RQ5 Can UID features detect subtle differences in text generation even when lexical and syntactic cues are indistinguishable?

Key findings

• GPT-who outperforms state-of-the-art statistical detectors like GLTR, GPTZero, and ZeroGPT by over 20% in F1 score across all benchmark datasets, particularly excelling in cross-domain and cross-model detection.
• On the Deepfake Text in-the-wild dataset, GPT-who achieves an F1 score of 0.85 in the most challenging 'unseen model and unseen domain' setting, outperforming GLTR (0.74) and DetectGPT (0.57).
• In the ArguGPT dataset, GPT-who achieves an F1 score of 0.84, outperforming human experts (0.51) and matching RoBERTa fine-tuned models, despite not being fine-tuned.
• On the GPABenchmark dataset, GPT-who exceeds GPTZero, ZeroGPT, and OpenAI’s detector by over 40% in accuracy for detecting text from 7 highly advanced LLMs.
• Human-generated texts show more uneven and diverse information distribution than LLMs, and UID features reflect architectural similarities—models from the same family cluster closely in UID space.
• The method remains computationally efficient and interpretable, using only a pre-trained language model and logistic regression, without any fine-tuning or complex architecture.

Figure 2: GPT-who uses token probabilities of articles to extract UID-based features. A classifier then learns to map UID features to different authors, and identify the author of a new unseen article.