[Paper Review] Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning

Dayiheng Liu, Jie Fu+3 more|arXiv (Cornell University)|May 29, 2019

Generative Adversarial Networks and Image Synthesis41 references4 citations

TL;DR

This paper proposes a novel unsupervised text style transfer framework that revises sentences in continuous space using gradient-based optimization, eliminating the need for adversarial learning and explicit disentanglement. By leveraging a VAE, attribute predictors, and a content predictor, the method enables controllable, interpretable, and multi-attribute style transfer, outperforming five state-of-the-art methods on three benchmark tasks.

ABSTRACT

Typical methods for unsupervised text style transfer often rely on two key ingredients: 1) seeking the explicit disentanglement of the content and the attributes, and 2) troublesome adversarial learning. In this paper, we show that neither of these components is indispensable. We propose a new framework that utilizes the gradients to revise the sentence in a continuous space during inference to achieve text style transfer. Our method consists of three key components: a variational auto-encoder (VAE), some attribute predictors (one for each attribute), and a content predictor. The VAE and the two types of predictors enable us to perform gradient-based optimization in the continuous space, which is mapped from sentences in a discrete space, to find the representation of a target sentence with the desired attributes and preserved content. Moreover, the proposed method naturally has the ability to simultaneously manipulate multiple fine-grained attributes, such as sentence length and the presence of specific words, when performing text style transfer tasks. Compared with previous adversarial learning based methods, the proposed method is more interpretable, controllable and easier to train. Extensive experimental studies on three popular text style transfer tasks show that the proposed method significantly outperforms five state-of-the-art methods.

Motivation & Objective

• To address the limitations of adversarial learning in text style transfer, such as training instability and poor disentanglement.
• To eliminate the need for explicit content-attribute disentanglement in unsupervised text style transfer.
• To enable simultaneous control over multiple fine-grained attributes, such as sentiment, sentence length, and keyword presence.
• To improve interpretability and controllability by revising sentence representations via gradient optimization during inference.
• To achieve strong performance on text style transfer tasks without requiring parallel training data.

Proposed method

• A variational auto-encoder (VAE) maps discrete sentences into a smooth continuous latent space and reconstructs them from the latent representation.
• Attribute predictors are trained to predict specific attributes (e.g., sentiment, length) from the continuous latent representation.
• A content predictor predicts the Bag-of-Words (BoW) feature of the generated sentence to preserve content and enforce keyword inclusion.
• During inference, the latent representation of the input sentence is revised via gradient descent using losses from the attribute and content predictors.
• The optimization process iteratively adjusts the latent code to minimize attribute and content prediction errors, yielding a target sentence with desired style and preserved content.
• The method naturally supports multi-attribute control by combining multiple loss terms for different attributes in the optimization objective.

Experimental results

Research questions

• RQ1 Can text style transfer be achieved without adversarial learning or explicit disentanglement of content and style?
• RQ2 Can gradient-based revision in continuous space improve controllability and interpretability in text style transfer?
• RQ3 Can the proposed method simultaneously control multiple fine-grained attributes such as sentence length and keyword presence?
• RQ4 How does the method compare to state-of-the-art adversarial learning-based approaches in terms of performance and training stability?
• RQ5 Can the method maintain high fluency and content preservation while manipulating complex attribute combinations?

Key findings

• The proposed method significantly outperforms five state-of-the-art methods on three popular text style transfer benchmarks.
• On sentiment transfer, the method achieves 93.0% accuracy, 183.7 Len% score, and 66.6 Key% score when controlling sentiment, length, and keywords simultaneously.
• The method achieves 87.6% accuracy, 60.9 Len% score, and 63.0 Key% score when reducing sentence length, demonstrating robustness in challenging control scenarios.
• The model shows higher fluency and content preservation when increasing sentence length compared to decreasing it, due to the difficulty of length reduction under constraints.
• The method enables multiple diverse outputs through gradient tuning, enhancing interpretability and controllability.
• The ablation study confirms that the content predictor significantly improves content preservation, especially in complex attribute manipulation tasks.