[Paper Review] Text Style Transfer: A Review and Experimental Evaluation

Zhiqiang Hu, Roy Ka-Wei Lee+2 more|arXiv (Cornell University)|Oct 24, 2020

Topic Modeling15 citations

TL;DR

This paper presents a comprehensive review and large-scale experimental benchmark of 19 state-of-the-art text style transfer (TST) models on two public datasets, evaluating their performance across style transfer accuracy, content preservation, and fluency. It introduces a taxonomy of TST methods, highlights limitations in current evaluation metrics, and identifies key challenges and future research directions in unsupervised learning, multi-style transfer, and cross-lingual style transfer.

ABSTRACT

The stylistic properties of text have intrigued computational linguistics researchers in recent years. Specifically, researchers have investigated the Text Style Transfer (TST) task, which aims to change the stylistic properties of the text while retaining its style independent content. Over the last few years, many novel TST algorithms have been developed, while the industry has leveraged these algorithms to enable exciting TST applications. The field of TST research has burgeoned because of this symbiosis. This article aims to provide a comprehensive review of recent research efforts on text style transfer. More concretely, we create a taxonomy to organize the TST models and provide a comprehensive summary of the state of the art. We review the existing evaluation methodologies for TST tasks and conduct a large-scale reproducibility study where we experimentally benchmark 19 state-of-the-art TST algorithms on two publicly available datasets. Finally, we expand on current trends and provide new perspectives on the new and exciting developments in the TST field.

Motivation & Objective

• To provide a systematic taxonomy and classification of recent text style transfer (TST) models.
• To evaluate and compare 19 state-of-the-art TST algorithms using a large-scale reproducibility study on public datasets.
• To analyze existing evaluation methodologies and identify their limitations in measuring style transfer effectiveness.
• To explore open challenges in TST, including unsupervised learning, multi-style transfer, and cross-lingual applications.
• To guide future research by identifying promising directions such as reducing reliance on style labels and improving automatic evaluation metrics.

Proposed method

• The authors classify TST models into categories based on their architecture and training paradigm, including those using parallel data, disentangled representations, and adversarial training.
• They conduct a large-scale reproducibility study, re-implementing 19 SOTA TST models on two public datasets: the Amazon review dataset and the Yelp review dataset.
• Evaluation is performed using automatic metrics including style transfer accuracy, content preservation (e.g., BERTScore), fluency (e.g., perplexity), and transfer strength.
• The study employs a style classifier to measure transfer effectiveness and analyzes the trade-off between style transfer and content preservation.
• The authors explore alternative supervision signals such as semantic relatedness, fluency, and readability to reduce dependency on labeled style data.
• They also examine domain-aware and multi-attribute style transfer as extensions beyond binary-style transfer.

Experimental results

Research questions

• RQ1 How do different TST models compare in terms of style transfer accuracy, content preservation, and fluency across public benchmarks?
• RQ2 What are the key architectural and training differences among state-of-the-art TST models, and how do they impact performance?
• RQ3 To what extent do existing automatic evaluation metrics correlate with human judgments, and what are their limitations?
• RQ4 Can TST be effectively performed without parallel data or style labels, and what alternative supervision signals are viable?
• RQ5 What are the most promising future directions for advancing TST, including multi-style transfer and cross-lingual applications?

Key findings

• No single TST model outperforms all others across all evaluation metrics, indicating that different models excel in different aspects such as style transfer or content preservation.
• There is a strong inverse relationship between transfer strength and content preservation, suggesting a fundamental trade-off in model optimization.
• Existing automatic evaluation metrics, particularly style transfer accuracy, are limited by the performance of the style classifier used to measure them.
• Models relying on style-content disentanglement are being increasingly replaced by end-to-end approaches that do not require explicit disentanglement, indicating a shift in research trends.
• Unsupervised and weakly supervised methods that use fluency, semantic similarity, and readability as auxiliary signals show promise in reducing dependency on labeled style data.
• Cross-lingual TST remains underexplored, with most models focused on English, highlighting a significant research gap in multilingual style transfer.