[Paper Review] TURINGBENCH: A Benchmark Environment for Turing Test in the Age of Neural Text Generation

Adaku Uchendu, Zeyu Ma+3 more|arXiv (Cornell University)|Sep 27, 2021

Topic Modeling4 citations

TL;DR

TuringBench introduces the first comprehensive benchmark environment for evaluating machine-generated text in the context of the Turing Test and authorship attribution, featuring 200K samples across 20 text-generators (including GPT-3, GPT-2, and FAIR models), two core tasks—Turing Test and Authorship Attribution—and an online leaderboard. FAIR_wmt20 and GPT-3 achieve the highest human-likeness, with the lowest F1 scores (best performance) among state-of-the-art detection models.

ABSTRACT

Recent progress in generative language models has enabled machines to generate astonishingly realistic texts. While there are many legitimate applications of such models, there is also a rising need to distinguish machine-generated texts from human-written ones (e.g., fake news detection). However, to our best knowledge, there is currently no benchmark environment with datasets and tasks to systematically study the so-called "Turing Test" problem for neural text generation methods. In this work, we present the TuringBench benchmark environment, which is comprised of (1) a dataset with 200K human- or machine-generated samples across 20 labels {Human, GPT-1, GPT-2_small, GPT-2_medium, GPT-2_large, GPT-2_xl, GPT-2_PyTorch, GPT-3, GROVER_base, GROVER_large, GROVER_mega, CTRL, XLM, XLNET_base, XLNET_large, FAIR_wmt19, FAIR_wmt20, TRANSFORMER_XL, PPLM_distil, PPLM_gpt2}, (2) two benchmark tasks -- i.e., Turing Test (TT) and Authorship Attribution (AA), and (3) a website with leaderboards. Our preliminary experimental results using TuringBench show that FAIR_wmt20 and GPT-3 are the current winners, among all language models tested, in generating the most human-like indistinguishable texts with the lowest F1 score by five state-of-the-art TT detection models. The TuringBench is available at: https://turingbench.ist.psu.edu/

Motivation & Objective

• To address the urgent need for systematic evaluation of neural text generation in distinguishing human-written from machine-generated text.
• To create a standardized benchmark environment with diverse, high-quality datasets across multiple language models.
• To enable research on detecting AI-generated content through two core tasks: Turing Test (binary classification of human vs. machine) and Authorship Attribution (identification of specific model origin).
• To support the development of robust detection models by providing a public, scalable platform with leaderboards and real-world political news prompts.
• To highlight the limitations of current detection methods and the increasing difficulty of distinguishing machine-generated text from human writing due to stylistic convergence.

Proposed method

• The benchmark constructs a dataset of 200K news articles (10K titles, 20 labels) by prompting 19 neural text-generators to produce articles similar to human-written news from outlets like CNN and Washington Post.
• Each sample is labeled with its source: 1 human, 19 machine models (e.g., GPT-3, GPT-2, GROVER, FAIR, PPLM, etc.).
• Two benchmark tasks are defined: (1) Turing Test—binary classification of human vs. machine-generated text; (2) Authorship Attribution—multi-class classification to identify the specific model used.
• A public website hosts the dataset, leaderboards, and evaluation tools to encourage community participation and model comparison.
• Statistical and visualization analyses (e.g., PCA of LIWC psycho-linguistic features) are used to study stylistic similarities and differences across human and machine-generated texts.
• F1 scores are used to evaluate detection model performance on the Turing Test, with ablation studies on text length and model complexity.

Experimental results

Research questions

• RQ1 Which neural text-generators produce the most human-like text that is indistinguishable from human writing?
• RQ2 How well can state-of-the-art detection models perform on the Turing Test task across diverse language models?
• RQ3 Can traditional authorship attribution models effectively distinguish between 20 different neural text-generators based on stylistic features?
• RQ4 To what extent do text length and model architecture influence detection performance in Turing Test and authorship attribution tasks?
• RQ5 What are the limitations of current detection models in capturing subtle stylistic differences between human and machine-generated texts?

Key findings

• FAIR_wmt20 and GPT-3 produce the most human-like text, achieving the lowest F1 scores (best detection performance) among 19 models on the Turing Test, indicating highest indistinguishability.
• There is no clear correlation between text length (100–400 tokens) and detection performance, except for RoBERTa, which shows length-dependent performance.
• Traditional authorship attribution models (e.g., SVM, Random Forest) and even deep learning models fail to fully capture stylistic distinctions across 20 authors, indicating a need for more sophisticated models.
• Psycho-linguistic features (LIWC) show that human-written texts span a wide stylistic range, while machine-generated texts cluster tightly in feature space, suggesting limited stylistic diversity in models.
• The overlap in stylistic features across 20 authors implies that machine-generated texts are becoming increasingly similar to human writing, making detection more challenging.
• The results suggest that black-box detectors are insufficient, and future work must focus on explainable detection frameworks to identify subtle, nuanced patterns in neural text generation.