[Paper Review] Can Large Language Models Identify Authorship?

Baixiang Huang, Canyu Chen+1 more|arXiv (Cornell University)|Mar 13, 2024

Authorship Attribution and Profiling4 citations

TL;DR

This paper investigates the use of Large Language Models (LLMs) for zero-shot authorship verification and attribution without domain-specific fine-tuning. Using a novel Linguistically Informed Prompting (LIP) technique, LLMs like GPT-4 achieve state-of-the-art performance and enhanced explainability by identifying key linguistic features, outperforming traditional and BERT-based models in both tasks across multiple datasets.

ABSTRACT

The ability to accurately identify authorship is crucial for verifying content authenticity and mitigating misinformation. Large Language Models (LLMs) have demonstrated an exceptional capacity for reasoning and problem-solving. However, their potential in authorship analysis remains under-explored. Traditional studies have depended on hand-crafted stylistic features, whereas state-of-the-art approaches leverage text embeddings from pre-trained language models. These methods, which typically require fine-tuning on labeled data, often suffer from performance degradation in cross-domain applications and provide limited explainability. This work seeks to address three research questions: (1) Can LLMs perform zero-shot, end-to-end authorship verification effectively? (2) Are LLMs capable of accurately attributing authorship among multiple candidates authors (e.g., 10 and 20)? (3) Can LLMs provide explainability in authorship analysis, particularly through the role of linguistic features? Moreover, we investigate the integration of explicit linguistic features to guide LLMs in their reasoning processes. Our assessment demonstrates LLMs' proficiency in both tasks without the need for domain-specific fine-tuning, providing explanations into their decision making via a detailed analysis of linguistic features. This establishes a new benchmark for future research on LLM-based authorship analysis.

Motivation & Objective

• To evaluate the effectiveness of LLMs in zero-shot, end-to-end authorship verification without fine-tuning.
• To assess LLMs' accuracy in authorship attribution across multiple candidate authors (e.g., 10 and 20).
• To enhance explainability in authorship analysis by integrating linguistic features into LLM reasoning.
• To develop and validate a new prompting technique, Linguistically Informed Prompting (LIP), that guides LLMs to focus on stylometric and linguistic features.
• To establish a new benchmark for LLM-based authorship analysis in low-resource and cross-domain settings.

Proposed method

• Propose Linguistically Informed Prompting (LIP), a prompting framework that explicitly guides LLMs to analyze linguistic features such as tone, humor, and syntax.
• Apply LLMs directly to authorship verification and attribution tasks in a zero-shot, end-to-end manner, bypassing intermediate feature engineering.
• Use GPT-4 and GPT-3.5 as primary LLMs, comparing their performance against TF-IDF, BERT, RoBERTa, DeBERTa, and ELECTRA.
• Integrate linguistic feature detection into the LLM prompting process to improve decision transparency and focus on stylistic cues.
• Generate natural language explanations for authorship decisions, with visualizations (e.g., word clouds) to analyze model attention to linguistic terms.
• Make all model outputs, input texts, and analyses publicly available via a GitHub repository for reproducibility and benchmarking.

Experimental results

Research questions

• RQ1 Can LLMs perform zero-shot, end-to-end authorship verification effectively?
• RQ2 Are LLMs capable of accurately attributing authorship among multiple candidate authors (e.g., 10 and 20)?
• RQ3 How can LLMs provide explainability in authorship analysis, particularly through the role of linguistic features?
• RQ4 Does integrating explicit linguistic guidance improve LLM performance and decision transparency in authorship tasks?

Key findings

• GPT-4 with LIP outperforms all baseline models, including BERT-based models and TF-IDF, in both authorship verification and attribution tasks.
• The performance of LLMs remains stable even when the number of candidate authors increases from 10 to 20, unlike traditional models that show significant performance drops.
• Linguistic-guided prompting significantly improves the specificity and relevance of LLM explanations, with word clouds showing increased focus on high-level features like 'humor', 'sarcasm', and 'colloquial'.
• Linguistic features such as tone and register are consistently detected and emphasized in LIP-guided outputs, demonstrating the model’s ability to reason about stylistic nuances.
• The LIP method enhances explainability by grounding decisions in observable linguistic attributes, offering clearer and more interpretable justifications than black-box embeddings.
• The study establishes a new benchmark for LLM-based authorship analysis, demonstrating strong zero-shot generalization and robustness across datasets without domain-specific fine-tuning.