[Paper Review] TOPFORMER: Topology-Aware Authorship Attribution of Deepfake Texts with Diverse Writing Styles

Adaku Uchendu, Thái Hoàng Lê+1 more|arXiv (Cornell University)|Sep 22, 2023

Authorship Attribution and Profiling4 citations

TL;DR

This paper proposes TopRoBERTa, a hybrid deep learning model that integrates Topological Data Analysis (TDA) into RoBERTa to improve multi-class authorship attribution of deepfake texts. By extracting topological features from the model's pooled output, TopRoBERTa achieves up to a 7% increase in Macro F1 score on heterogeneous and imbalanced datasets, outperforming vanilla RoBERTa and Gaussian-augmented baselines.

ABSTRACT

Recent advances in Large Language Models (LLMs) have enabled the generation of open-ended high-quality texts, that are non-trivial to distinguish from human-written texts. We refer to such LLM-generated texts as deepfake texts. There are currently over 72K text generation models in the huggingface model repo. As such, users with malicious intent can easily use these open-sourced LLMs to generate harmful texts and dis/misinformation at scale. To mitigate this problem, a computational method to determine if a given text is a deepfake text or not is desired--i.e., Turing Test (TT). In particular, in this work, we investigate the more general version of the problem, known as Authorship Attribution (AA), in a multi-class setting--i.e., not only determining if a given text is a deepfake text or not but also being able to pinpoint which LLM is the author. We propose TopFormer to improve existing AA solutions by capturing more linguistic patterns in deepfake texts by including a Topological Data Analysis (TDA) layer in the Transformer-based model. We show the benefits of having a TDA layer when dealing with imbalanced, and multi-style datasets, by extracting TDA features from the reshaped $pooled\_output$ of our backbone as input. This Transformer-based model captures contextual representations (i.e., semantic and syntactic linguistic features), while TDA captures the shape and structure of data (i.e., linguistic structures). Finally, TopFormer, outperforms all baselines in all 3 datasets, achieving up to 7\% increase in Macro F1 score. Our code and datasets are available at: https://github.com/AdaUchendu/topformer

Motivation & Objective

• Address the growing challenge of detecting and attributing deepfake texts generated by large language models (LLMs), which are increasingly difficult to distinguish from human-written content.
• Extend beyond binary Turing Test classification to multi-class authorship attribution, enabling identification of the specific LLM responsible for generating a given text.
• Improve robustness in real-world scenarios involving noisy, imbalanced, and heterogeneous text data, which are common in LLM-generated text detection.
• Develop a hybrid model that combines the representational power of RoBERTa with the structural sensitivity of Topological Data Analysis (TDA) to capture subtle linguistic patterns in deepfake texts.
• Demonstrate that TDA features extracted from the pooled output of RoBERTa enhance model generalization and performance, especially in complex, non-uniform data distributions.

Proposed method

• Fine-tune RoBERTa-base to extract contextual embeddings (semantic and syntactic features) from input texts, using its pooled_output as the primary representation.
• Apply Topological Data Analysis (TDA) to the reshaped pooled_output to extract topological features that capture the underlying shape and structure of linguistic patterns in the embedding space.
• Concatenate the TDA features with the original RoBERTa embeddings before feeding them into a final classification head, effectively augmenting the model’s input with topological invariants.
• Use persistent homology to compute TDA features from the pooled_output, enabling the model to detect structural patterns resistant to noise and distribution shifts.
• Normalize TDA feature vectors across samples to ensure consistent dimensionality and avoid instability, especially when feature counts vary between sequences.
• Compare the performance of TopRoBERTa against baseline models (e.g., vanilla RoBERTa, Gaussian-RoBERTa) to validate the contribution of the TDA layer under varying data conditions.

Experimental results

Research questions

• RQ1 Can the integration of Topological Data Analysis (TDA) into RoBERTa improve authorship attribution performance for deepfake texts in noisy and imbalanced datasets?
• RQ2 Does the TDA layer enhance model robustness when classifying texts from diverse sources, such as paraphrasers, translators, and multiple LLMs?
• RQ3 Is the performance gain from the TDA layer due to inherent model capacity or due to data-specific noise patterns, particularly in heterogeneous data distributions?
• RQ4 How does using the pooled_output as input for TDA compare to using attention weights in terms of stability, computational cost, and classification accuracy?
• RQ5 To what extent does TopRoBERTa generalize to out-of-distribution and low-resource language settings, especially under adversarial conditions?

Key findings

• TopRoBERTa outperforms vanilla RoBERTa on 2 out of 3 datasets, achieving a maximum 7% increase in Macro F1 score, particularly on heterogeneous and multi-domain data.
• The model shows superior performance on the SynSciPass dataset (with 12 labels: humans, generators, translators, paraphrasers), where it achieves a 7% F1 improvement, indicating strong generalization to complex label distributions.
• TopRoBERTa consistently outperforms Gaussian-augmented baselines (Gaussian-BERT and Gaussian-RoBERTa), suggesting that the performance gain is not due to random noise adaptation but to meaningful topological feature extraction.
• TDA features extracted from the pooled_output yield more stable and informative representations than those derived from attention weights, which exhibit high variance and require additional preprocessing.
• Visualization of PCA-reduced embeddings shows more distinct and structured clusters in TopRoBERTa’s output, confirming that TDA enhances feature separability in high-dimensional space.
• The TDA layer provides minimal benefit on homogeneous datasets (e.g., Human vs. Paraphraser), indicating its primary advantage lies in handling heterogeneous and complex label structures.