[Paper Review] A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer

Fuli Luo, Peng Li+5 more|arXiv (Cornell University)|May 24, 2019

Generative Adversarial Networks and Image Synthesis27 references40 citations

TL;DR

Introduces DualRL, a dual reinforcement learning approach that directly learns one-step source-to-target and target-to-source mappings for unsupervised text style transfer, guided by style-accuracy and content-preservation rewards, without relying on parallel data.

ABSTRACT

Unsupervised text style transfer aims to transfer the underlying style of text but keep its main content unchanged without parallel data. Most existing methods typically follow two steps: first separating the content from the original style, and then fusing the content with the desired style. However, the separation in the first step is challenging because the content and style interact in subtle ways in natural language. Therefore, in this paper, we propose a dual reinforcement learning framework to directly transfer the style of the text via a one-step mapping model, without any separation of content and style. Specifically, we consider the learning of the source-to-target and target-to-source mappings as a dual task, and two rewards are designed based on such a dual structure to reflect the style accuracy and content preservation, respectively. In this way, the two one-step mapping models can be trained via reinforcement learning, without any use of parallel data. Automatic evaluations show that our model outperforms the state-of-the-art systems by a large margin, especially with more than 8 BLEU points improvement averaged on two benchmark datasets. Human evaluations also validate the effectiveness of our model in terms of style accuracy, content preservation and fluency. Our code and data, including outputs of all baselines and our model are available at https://github.com/luofuli/DualLanST.

Motivation & Objective

• Motivate unsupervised text style transfer without parallel data by avoiding explicit content-style separation.
• Propose a dual one-step mapping framework for bidirectional style transfer (X->Y and Y->X).
• Design dual rewards that quantify style accuracy and content preservation to guide training via reinforcement learning.
• Introduce annealing pseudo teacher-forcing to bootstrap training with pseudo-parallel data and gradually shift to pure RL training.

Proposed method

• Model two one-step seq2seq mappings: f_theta: X->Y and g_phi: Y->X.
• Use a style classifier to provide a style accuracy reward R_s for transferred outputs.
• Use backward reconstruction probability P(x|y') as a content preservation reward R_c.
• Combine rewards into a harmonic overall reward R to train f and g via policy gradient (Equation 4).
• Employ annealing pseudo teacher-forcing to create pseudo-parallel data and gradually rely on RL, with schedule in Equation 5.
• Pre-train with pseudo-parallel data, then alternate RL updates between f and g to form a closed loop (Algorithm 1).
• Use back-translation-inspired pseudo-data generation and an annealing schedule to mitigate exposure bias (Algorithm 2).
• Evaluate with automatic metrics (style accuracy, BLEU, G2, H2) and human judgments on Yelp (sentiment) and Gyafc (formality).

Experimental results

Research questions

• RQ1 Can a dual one-step mapping framework learn unsupervised text style transfer without explicit content-style separation?
• RQ2 Do dual rewards for style accuracy and content preservation enable effective RL-based training with non-parallel data?
• RQ3 Does annealing pseudo teacher-forcing help bootstrap RL and reduce reliance on pseudo-parallel data over time?
• RQ4 How does DualRL compare to state-of-the-art unsupervised style-transfer methods in automatic and human evaluation?

Key findings

• DualRL achieves best overall performance in automatic evaluation (G2 and H2) across two datasets, with large improvements in BLEU-based content preservation metrics.
• Human evaluations show DualRL yields the best average scores and a higher rate of successful transfers compared to baselines.
• The approach balances style accuracy, content preservation, and fluency, outperforming several strong baselines on both sentiment (Yelp) and formality (Gyafc) tasks.
• Ablation studies reveal that combining RL with MLE (RL+MLE) offers the best trade-off between style accuracy and content/fluency.
• The method demonstrates that directly learning bidirectional one-step mappings with dual rewards can surpass prior two-step or back-translation-based approaches.