[Paper Review] PART: Pre-trained Authorship Representation Transformer

Javier Huertas‐Tato, Alejandro Martı́n+1 more|arXiv (Cornell University)|Sep 30, 2022

Authorship Attribution and Profiling4 citations

TL;DR

This paper proposes PART (Pre-trained Authorship Representation Transformer), a contrastive self-supervised learning model that generates zero-shot authorship embeddings by training on pairs of documents from the same author. It achieves 72.39% zero-shot accuracy and 86.73% top-5 accuracy on author identification across 250 unseen authors, demonstrating strong generalization to out-of-domain writing styles using a heterogeneous training set of literature, blogs, and emails.

ABSTRACT

Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. Using stylometric representations is more suitable, but this by itself is an open research challenge. In this paper, we propose PART, a contrastively trained model fit to learn extbf{authorship embeddings} instead of semantics. We train our model on ~1.5M texts belonging to 1162 literature authors, 17287 blog posters and 135 corporate email accounts; a heterogeneous set with identifiable writing styles. We evaluate the model on current challenges, achieving competitive performance. We also evaluate our model on test splits of the datasets achieving zero-shot 72.39\% accuracy when bounded to 250 authors, a 54\% and 56\% higher than RoBERTa embeddings. We qualitatively assess the representations with different data visualizations on the available datasets, observing features such as gender, age, or occupation of the author.

Motivation & Objective

• To address the limitations of hand-crafted features and supervised models in authorship attribution by learning generalizable, style-aware representations.
• To develop a zero-shot capable model that generalizes to out-of-domain authors without fine-tuning.
• To generate consistent authorship embeddings for words, sentences, and documents using a unified representation space.
• To evaluate the model’s ability to capture stylistic features such as gender, age, occupation, and genre through qualitative and quantitative analysis.
• To investigate the impact of training data diversity and label bias on representation quality and model generalization.

Proposed method

• The model uses a pre-trained Transformer encoder with an LSTM head to process document pairs.
• Contrastive learning is applied by maximizing cosine similarity between embeddings of documents from the same author while minimizing similarity with documents from different authors.
• The loss function is based on the InfoNCE contrastive objective, optimizing for intra-author similarity and inter-author dissimilarity.
• Document-level embeddings are obtained via average pooling of token-level representations, enabling consistent representation across varying text lengths.
• The model is pre-trained on a diverse corpus spanning literature, anonymous blogs, and corporate emails to capture heterogeneous writing styles.
• The final authorship embedding is a learned vector representation that captures stylistic idiosyncrasies independent of semantic content.

Experimental results

Research questions

• RQ1 Can contrastive self-supervised learning effectively learn authorship-specific representations without explicit labels?
• RQ2 To what extent can the model generalize to unseen authors and writing styles not present in the training data?
• RQ3 How do data biases in the training set affect the quality and generalization of learned authorship embeddings?
• RQ4 Can the learned embeddings capture meaningful stylistic attributes such as gender, age, occupation, and genre?
• RQ5 How does the model perform under zero-shot evaluation across diverse authorship datasets with varying data quality and representation?

Key findings

• The model achieves 72.39% zero-shot accuracy and 86.73% top-5 accuracy on a joint evaluation dataset with 250 unseen authors, demonstrating strong zero-shot generalization.
• Qualitative analysis confirms that embeddings cluster by author and stylistic features such as gender, age, occupation, and genre, indicating meaningful representation learning.
• The model shows high consistency in document-level embeddings across varying text lengths due to average pooling of token representations.
• Despite high performance, the model exhibits bias toward over-represented labels in training, such as fiction in literature and student status in blogs, indicating data representation impacts embedding quality.
• The model’s performance is limited by sequence length, as longer context is needed to capture long-range stylistic dependencies.
• The authors identify that more diverse training data—especially from underrepresented domains like scientific journals and essays—is critical to improving robustness and generalization.