[Paper Review] Stylometry Analysis of Multi-authored Documents for Authorship and Author Style Change Detection

Muhammad Tayyab Zamir, Muhammad Asif Ayub+3 more|arXiv (Cornell University)|Jan 12, 2024

Authorship Attribution and Profiling4 citations

TL;DR

This paper proposes a merit-based late fusion framework combining multiple transformer models and optimization techniques (PSO, Nelder-Mead, Powell) for stylometry tasks in multi-authored documents. It demonstrates that retaining special characters during pre-processing significantly improves performance, achieving up to 5.35% higher F1-score on multiple author-switching detection compared to state-of-the-art methods.

ABSTRACT

In recent years, the increasing use of Artificial Intelligence based text generation tools has posed new challenges in document provenance, authentication, and authorship detection. However, advancements in stylometry have provided opportunities for automatic authorship and author change detection in multi-authored documents using style analysis techniques. Style analysis can serve as a primary step toward document provenance and authentication through authorship detection. This paper investigates three key tasks of style analysis: (i) classification of single and multi-authored documents, (ii) single change detection, which involves identifying the point where the author switches, and (iii) multiple author-switching detection in multi-authored documents. We formulate all three tasks as classification problems and propose a merit-based fusion framework that integrates several state-of-the-art natural language processing (NLP) algorithms and weight optimization techniques. We also explore the potential of special characters, which are typically removed during pre-processing in NLP applications, on the performance of the proposed methods for these tasks by conducting extensive experiments on both cleaned and raw datasets. Experimental results demonstrate significant improvements over existing solutions for all three tasks on a benchmark dataset.

Motivation & Objective

• Address the growing challenge of authorship attribution and style change detection in AI-generated and multi-authored documents.
• Improve performance on three stylometry tasks: single/multi-authored document classification, single author-switch detection, and multiple author-switch detection.
• Investigate the impact of retaining special characters—commonly removed during NLP pre-processing—on stylometric analysis performance.
• Develop a merit-based fusion framework that optimizes model weights using advanced optimization techniques to enhance classification accuracy.
• Extend prior work by addressing both single and multiple author-switching detection in real-world multi-authored documents.

Proposed method

• Formulate all three stylometry tasks as text classification problems using transformer-based models (e.g., BERT, RoBERTa, DistilBERT).
• Implement a late fusion strategy where predictions from individual models are combined using weighted averaging.
• Optimize model weights using three meta-heuristic optimization methods: Particle Swarm Optimization (PSO), Nelder-Mead, and Powell’s method.
• Conduct experiments on both cleaned and raw datasets to evaluate the impact of pre-processing, particularly the retention of stop words, contractions, and punctuation.
• Use the F1-score as the primary evaluation metric across all tasks, comparing results against existing SOTA solutions.
• Apply feature engineering to preserve stylistic markers such as special characters, which are typically discarded in standard NLP pipelines.

Figure 1: Single and multi-authored document samples: (a) single-authored document, (b) multi-authored document zangerle2021overview .

Experimental results

Research questions

• RQ1 Can a merit-based late fusion framework combining multiple transformer models improve performance on stylometry tasks compared to individual models?
• RQ2 How do different optimization techniques (PSO, Nelder-Mead, Powell) affect the performance of fused models in authorship and style change detection?
• RQ3 To what extent do special characters and stop/short words contribute to distinguishing authorial style in multi-authored documents?
• RQ4 Does retaining raw text features (e.g., punctuation, contractions) during pre-processing lead to measurable improvements in stylometric classification tasks?
• RQ5 How does the performance of the proposed method vary across different stylometry tasks, especially in detecting multiple author-switching events?

Key findings

• The PSO- and Nelder-Mead-based fusion methods achieved the highest F1-score of 0.5535 on the multiple author-switching detection task (task 3) using the cleaned dataset, outperforming existing solutions by approximately 5%.
• On task 2 (single author-switch detection), the simple fusion method achieved an F1-score of 0.782, representing a 3% improvement over the best existing solution.
• The inclusion of special characters and stop words significantly improved model performance, as their removal led to a notable drop in F1-scores across all tasks.
• The Nelder-Mead and PSO-based fusion methods outperformed individual models and other fusion strategies, especially on complex tasks involving multiple author switches.
• The performance gap between clean and raw datasets was minimal on task 3, indicating that the proposed method is robust to pre-processing variations when special characters are preserved.
• The BERT-base model achieved the highest individual performance on task 3 (F1 = 0.5309 on clean data), but fusion methods surpassed it, demonstrating the effectiveness of ensemble learning in stylometry.

Figure 2: A sample multi-authored document with a single author switching zangerle2021overview . The red line indicates the position where the author switches from author X to author Y.