[Paper Review] ST$^2$: Small-data Text Style Transfer via Multi-task Meta-Learning

Xiwen Chen, Kenny Q. Zhu|arXiv (Cornell University)|Apr 24, 2020

Topic Modeling25 references4 citations

TL;DR

This paper proposes ST², a multi-task meta-learning framework for few-shot text style transfer that enhances performance on low-resource styles—such as personal writing styles—by leveraging cross-style knowledge. By adapting meta-learning to base models like CrossAlign and VAE, ST² improves content preservation, style transfer accuracy, and fluency even with minimal training data, outperforming state-of-the-art methods on both a novel literature-style dataset and standard benchmarks.

ABSTRACT

Text style transfer aims to paraphrase a sentence in one style into another style while preserving content. Due to lack of parallel training data, state-of-art methods are unsupervised and rely on large datasets that share content. Furthermore, existing methods have been applied on very limited categories of styles such as positive/negative and formal/informal. In this work, we develop a meta-learning framework to transfer between any kind of text styles, including personal writing styles that are more fine-grained, share less content and have much smaller training data. While state-of-art models fail in the few-shot style transfer task, our framework effectively utilizes information from other styles to improve both language fluency and style transfer accuracy.

Motivation & Objective

• To address the challenge of few-shot text style transfer, especially for low-resource, fine-grained writing styles such as personal authorial voices.
• To overcome the failure of existing state-of-the-art models on small training datasets, which often lack sufficient content overlap.
• To develop a meta-learning framework that transfers knowledge across multiple style pairs to improve performance on individual few-shot tasks.
• To create and release the first publicly available literature-style transfer dataset for benchmarking few-shot style transfer.
• To demonstrate that meta-learning enhances not only model generalization but also disentanglement of style and content in latent space.

Proposed method

• Adapts model-agnostic meta-learning (MAML) to text style transfer, enabling rapid adaptation to new style pairs with few examples.
• Uses two base models—CrossAlign and VAE—as the backbone, both of which disentangle content and style in latent space.
• Trains on multiple style transfer tasks simultaneously, where each task corresponds to a unique style pair (e.g., formal-to-informal, author A-to-author B).
• Leverages off-domain data from other style pairs during meta-training to improve zero-shot and few-shot generalization.
• Applies meta-optimization to update model parameters such that fast adaptation is possible with only a few gradient steps on a new style pair.
• Pre-trains base models on all available data before meta-training to eliminate bias from insufficient data in individual tasks.

Experimental results

Research questions

• RQ1 Can meta-learning effectively improve few-shot text style transfer performance when training data per style pair is extremely limited?
• RQ2 Does knowledge transfer across different style pairs enhance content preservation, style transfer accuracy, and fluency in low-resource settings?
• RQ3 How does the proposed ST² framework compare to state-of-the-art models in terms of disentanglement of style and content in the latent space?
• RQ4 To what extent does meta-learning improve model generalization beyond just language modeling, especially when base models struggle on small datasets?
• RQ5 Can the framework be effectively applied to fine-grained, personal writing styles that are not covered by existing large-scale datasets?

Key findings

• ST² significantly outperforms state-of-the-art models in content preservation, style transfer accuracy, and fluency on both the novel literature-style dataset and standard benchmarks.
• The proposed framework improves performance even when base models fail to learn effective language models from small datasets, indicating that meta-learning provides a substantial boost beyond pretraining.
• t-SNE visualization confirms that ST² achieves better disentanglement of style and content embeddings in the latent space compared to pretrained base models.
• The model achieves strong performance on personal writing styles with minimal training data, demonstrating its applicability to fine-grained, low-resource style transfer.
• The ablation study shows that meta-learning improves both the language model and the style transfer capability, not just one component.
• The release of the literature-style transfer dataset provides a new benchmark for evaluating few-shot style transfer in realistic, low-resource settings.