[Paper Review] StyloAI: Distinguishing AI-Generated Content with Stylometric Analysis

Chidimma Opara|arXiv (Cornell University)|May 16, 2024

Biomedical Text Mining and Ontologies4 citations

TL;DR

StyloAI proposes a data-driven, interpretable model using 31 stylometric features—12 newly identified for AI text detection—trained on a Random Forest classifier to distinguish AI-generated from human-written text. It achieves 81% accuracy on the AuTextification dataset and 98% on an education-focused dataset, outperforming state-of-the-art models while offering transparency over 'black box' deep learning approaches.

ABSTRACT

The emergence of large language models (LLMs) capable of generating realistic texts and images has sparked ethical concerns across various sectors. In response, researchers in academia and industry are actively exploring methods to distinguish AI-generated content from human-authored material. However, a crucial question remains: What are the unique characteristics of AI-generated text? Addressing this gap, this study proposes StyloAI, a data-driven model that uses 31 stylometric features to identify AI-generated texts by applying a Random Forest classifier on two multi-domain datasets. StyloAI achieves accuracy rates of 81% and 98% on the test set of the AuTextification dataset and the Education dataset, respectively. This approach surpasses the performance of existing state-of-the-art models and provides valuable insights into the differences between AI-generated and human-authored texts.

Motivation & Objective

• To identify stylometric features that reliably differentiate AI-generated text from human-authored text across diverse domains.
• To develop a machine learning model that leverages these features for accurate, interpretable text classification.
• To address the limitations of 'black box' deep learning models in AI text detection by emphasizing feature interpretability.
• To evaluate the model's generalization capability across multi-domain datasets, particularly in education contexts.
• To provide a scalable, data-driven alternative to proprietary AI detectors with improved performance and transparency.

Proposed method

• The model extracts 31 stylometric features grouped into six categories: Lexical Diversity, Sentiment and Subjectivity, Readability, Named Entities, and Uniqueness and Variety.
• Among these, 12 features are newly proposed for AI text detection, including Type-Token Ratio (TTR), Hapax Legomenon Rate (HLR), UniqueWordCount, and StopWordCount.
• A Random Forest classifier is trained on these features to classify text as AI-generated or human-written.
• The model is evaluated on two multi-domain, annotated datasets: AuTextification (Bloom-generated) and an education-specific dataset.
• Feature importance is analyzed using the Random Forest's built-in permutation method to identify the most influential features.
• Performance is benchmarked against state-of-the-art models, including TALN-UPF, BOW+LR, and GPTZero, using F1-score and accuracy metrics.

Experimental results

Research questions

• RQ1 What stylometric features most effectively differentiate AI-generated text from human-authored text across multiple domains?
• RQ2 Can a transparent, non-deep learning model outperform existing state-of-the-art AI text detection systems in accuracy and generalization?
• RQ3 How do the most influential stylometric features reflect the linguistic patterns unique to AI-generated content?
• RQ4 To what extent does the model generalize across different text domains, particularly in education settings?
• RQ5 How does the interpretability of feature-based models compare to 'black box' deep learning approaches in AI text detection?

Key findings

• StyloAI achieves 81% accuracy on the AuTextification dataset, outperforming the TALN-UPF model (80% F1-score) and significantly surpassing the zero-shot baseline (33% F1-score).
• On the education-focused dataset, StyloAI achieves 98% accuracy, matching the performance of Mindner et al.'s model while using only handcrafted features.
• The top four most influential features are UniqueWordCount, StopWordCount, Type-Token Ratio (TTR), and HapaxLegomenonRate, indicating their critical role in distinguishing AI from human writing.
• AI-generated texts tend to overuse rare words and underuse stop words, disrupting the natural lexical balance found in human writing.
• The model demonstrates strong generalization across domains, particularly excelling in educational text detection, where it matches or exceeds the performance of models relying on fine-tuned transformers or proprietary detectors.
• The study confirms that handcrafted stylometric features combined with a transparent classifier like Random Forest can achieve state-of-the-art performance without requiring domain-specific fine-tuning or large-scale pretraining.