[Paper Review] Rethinking the Authorship Verification Experimental Setups

Florin Brad, Andrei Manolache+4 more|arXiv (Cornell University)|Dec 9, 2021

Authorship Attribution and Profiling4 citations

TL;DR

This paper proposes five new, progressively more challenging splits of the PAN authorship verification dataset to address data leakage and generalization issues in existing benchmarks. By isolating topic and stylistic biases and replacing named entities with placeholders, BERT-based models achieve improved generalization—especially on the out-of-domain DarkReddit dataset—demonstrating that models relying on spurious named entity cues perform poorly under distribution shift.

ABSTRACT

One of the main drivers of the recent advances in authorship verification is the PAN large-scale authorship dataset. Despite generating significant progress in the field, inconsistent performance differences between the closed and open test sets have been reported. To this end, we improve the experimental setup by proposing five new public splits over the PAN dataset, specifically designed to isolate and identify biases related to the text topic and to the author's writing style. We evaluate several BERT-like baselines on these splits, showing that such models are competitive with authorship verification state-of-the-art methods. Furthermore, using explainable AI, we find that these baselines are biased towards named entities. We show that models trained without the named entities obtain better results and generalize better when tested on DarkReddit, our new dataset for authorship verification.

Motivation & Objective

• Address inconsistent performance between closed and open test sets in PAN authorship verification benchmarks.
• Isolate and mitigate biases related to text topic and author writing style in existing experimental setups.
• Improve model generalization by reducing reliance on spurious features such as named entities.
• Introduce a new out-of-domain dataset, DarkReddit, to evaluate cross-corpus generalization capabilities.
• Provide publicly available, reproducible splits and baselines to support large-scale model evaluation and future research.

Proposed method

• Design five new dataset splits—Closed, Clopen, Open Unseen Authors, Open Unseen Fandoms, and Open All—ranging from closed-set to fully open-set configurations.
• Apply BERT-based models as baselines, fine-tuned on training splits and evaluated on the five test splits to assess generalization.
• Use Integrated Gradients for explainable AI (XAI) to analyze model attention and identify reliance on named entities.
• Replace named entities in training data with placeholders (e.g., [PERSON], [LOCATION]) to reduce spurious feature learning.
• Train and evaluate models on both PAN and the newly introduced DarkReddit dataset to test zero-shot generalization under distribution shift.
• Aggregate predictions across 512-token sequence chunks for long documents, handling context length limitations of BERT.

Experimental results

Research questions

• RQ1 To what extent do existing PAN dataset splits introduce data leakage that inflates performance on closed-set evaluations?
• RQ2 How do BERT-based models perform across increasingly open experimental setups, and do they generalize better when trained without named entities?
• RQ3 What role do named entities play in authorship verification decisions, and can their removal improve model robustness?
• RQ4 How well do models trained on PAN generalize to a significantly different domain, such as DarkReddit?
• RQ5 Can a simple data preprocessing step—replacing named entities with placeholders—lead to measurable performance gains and better generalization?

Key findings

• BERT-based models achieve competitive performance, scoring 97.4% on the Clopen split and 91.6% on the Open UF split, outperforming non-neural baselines and approaching SOTA O2D2 results.
• Models trained with named entities removed show a 5.6% improvement in F1 score on the DarkReddit test set, confirming better generalization under distribution shift.
• Integrated Gradients analysis reveals that models heavily rely on named entities for authorship decisions, especially in open-set scenarios.
• Replacing named entities with placeholders improves overall performance by 1.4 points (from 81.7 to 83.1) on the Open All split, with AUC increasing from 91.1 to 94.3.
• The O2D2 model performs poorly on the most challenging Open All split, while BERT models trained without named entities generalize significantly better.
• Performance on the development set is more predictive of test set performance when named entities are removed, indicating reduced overfitting to spurious features.