[Paper Review] Assessing the Stylistic Properties of Neurally Generated Text in Authorship Attribution

Enrique Manjavacas, Jeroen De Gussem+2 more|arXiv (Cornell University)|Aug 18, 2017

Authorship Attribution and Profiling20 references3 citations

TL;DR

This paper evaluates the stylistic fidelity of neurally generated text for authorship attribution using recurrent neural language models (RNNLM) and n-gram language models (NGLM). It finds that while RNNLM-generated text is fuzzier and harder to attribute than NGLM output, it significantly improves performance in data augmentation scenarios by introducing stylistically nuanced, diverse training examples that enhance classifier generalization beyond real data alone.

ABSTRACT

Recent applications of neural language models have led to an increased interest in the automatic generation of natural language. However impressive, the evaluation of neurally generated text has so far remained rather informal and anecdotal. Here, we present an attempt at the systematic assessment of one aspect of the quality of neurally generated text. We focus on a specific aspect of neural language generation: its ability to reproduce authorial writing styles. Using established models for authorship attribution, we empirically assess the stylistic qualities of neurally generated text. In comparison to conventional language models, neural models generate fuzzier text that is relatively harder to attribute correctly. Nevertheless, our results also suggest that neurally generated text offers more valuable perspectives for the augmentation of training data.

Motivation & Objective

• To systematically assess the stylistic properties of neurally generated text in the context of authorship attribution.
• To compare the performance of neural language models (RNNLM) versus traditional n-gram language models (NGLM) in preserving authorial style.
• To evaluate whether neurally generated text can serve as effective training data augmentation for authorship attribution, especially for authors with limited reference data.
• To investigate the trade-off between stylistic fidelity and diversity in generated text and its impact on classifier performance.

Proposed method

• Text generation via character-level language modeling using both RNNLM and NGLM architectures.
• Sampling from the language model’s output distribution using multinomial sampling with temperature control to balance diversity and fluency.
• Training authorship attribution classifiers on real text, generated text, and combined real+generated data to evaluate performance.
• Using normalized n-gram frequencies (orders 2–4) as stylistic features for document vectorization and PCA visualization.
• Applying Jaccard similarity to measure lexical overlap between authentic and generated texts across multiple authors.
• Employing principal component analysis (PCA) to visualize stylistic distribution shifts between real and generated text samples.

Experimental results

Research questions

• RQ1 To which extent is text generated from a single author’s oeuvre still attributable to the original author using standard authorship attribution methods?
• RQ2 How does the stylistic quality of RNNLM-generated text compare to NGLM-generated text in terms of similarity to authentic source material?
• RQ3 In what ways does incorporating neurally generated text improve authorship attribution performance when used for training data augmentation?
• RQ4 To what extent do generated texts preserve lexical and syntactic features of the original author’s style, as measured by n-gram overlap and distributional similarity?

Key findings

• RNNLM-generated text exhibits a significantly different n-gram frequency distribution compared to authentic text, indicating stylistic fuzziness and reduced direct attribution accuracy.
• NGLM-generated text closely mirrors the n-gram frequency distribution of the original corpus, resulting in higher attribution accuracy when used in isolation.
• In data augmentation settings, RNNLM-generated data leads to a measurable increase in attribution performance compared to real data alone, outperforming NGLM-generated data in this context.
• Jaccard similarity analysis shows that NGLM-generated texts have a much higher lexical overlap with authentic texts than RNNLM-generated texts, confirming the latter’s greater stylistic divergence.
• PCA visualizations confirm that RNNLM-generated data occupies a broader, more dispersed region in stylistic space, suggesting greater diversity and complexity.
• The results suggest that the 'fuzziness' of RNNLM output introduces valuable stylistic variation that enhances classifier generalization when used for data augmentation.