[Paper Review] A Probabilistic Formulation of Unsupervised Text Style Transfer

Junxian He, Xinyi Wang+2 more|arXiv (Cornell University)|Feb 10, 2020

Topic Modeling32 references63 citations

TL;DR

The paper proposes a deep latent sequence model for unsupervised text style transfer that treats non-parallel corpora as a partially observed parallel corpus, using amortized variational inference to train an encoder–decoder system with a language-model prior, achieving state-of-the-art results across multiple style transfer tasks and competitive unsupervised machine translation performance.

ABSTRACT

We present a deep generative model for unsupervised text style transfer that unifies previously proposed non-generative techniques. Our probabilistic approach models non-parallel data from two domains as a partially observed parallel corpus. By hypothesizing a parallel latent sequence that generates each observed sequence, our model learns to transform sequences from one domain to another in a completely unsupervised fashion. In contrast with traditional generative sequence models (e.g. the HMM), our model makes few assumptions about the data it generates: it uses a recurrent language model as a prior and an encoder-decoder as a transduction distribution. While computation of marginal data likelihood is intractable in this model class, we show that amortized variational inference admits a practical surrogate. Further, by drawing connections between our variational objective and other recent unsupervised style transfer and machine translation techniques, we show how our probabilistic view can unify some known non-generative objectives such as backtranslation and adversarial loss. Finally, we demonstrate the effectiveness of our method on a wide range of unsupervised style transfer tasks, including sentiment transfer, formality transfer, word decipherment, author imitation, and related language translation. Across all style transfer tasks, our approach yields substantial gains over state-of-the-art non-generative baselines, including the state-of-the-art unsupervised machine translation techniques that our approach generalizes. Further, we conduct experiments on a standard unsupervised machine translation task and find that our unified approach matches the current state-of-the-art.

Motivation & Objective

• Motivate a principled probabilistic approach to unsupervised text style transfer that unifies existing non-generative methods.
• Treat non-parallel corpora from two domains as a partially observed parallel corpus to enable learning of cross-domain transductions.
• Develop an encoder–decoder transduction model with a pretrained language-model prior and amortized variational inference to optimize a tractable surrogate objective (ELBO).
• Investigate the connections between the probabilistic objective and backtranslation/adversarial-style objectives and assess performance across diverse style-transfer tasks.
• Demonstrate that the proposed approach yields substantial gains over state-of-the-art non-generative baselines and matches or surpasses unsupervised MT baselines on relevant tasks.

Proposed method

• Define a deep latent sequence model for bitext with latent parallel sentences linking domains.
• Use two transduction distributions p(x|bar{y}) and p(y|bar{x}) parameterized by encoder–decoder modules, and domain-specific pretrained language priors.
• Apply amortized variational inference to maximize the ELBO with inference networks q(bar{y}|x) and q(bar{x}|y) that share parameters with the generative models.
• Tie parameters across the two transfer directions and incorporate a domain-embedding mechanism to specify transfer direction.
• Address intractable marginal likelihood via stochastic gradient estimates (with a stop-gradient approach for latent reconstruction) and employ Gumbel-Softmax or REINFORCE variants as needed, favoring greedy decoding for reconstruction.
• Initialization uses a self-reconstruction loss to stabilize early training and prevent poor local optima.

Experimental results

Research questions

• RQ1 Can a fully probabilistic formulation unify and improve unsupervised text style transfer approaches over existing non-generative methods?
• RQ2 Does amortized variational inference with latent bitext offer a practical and effective objective for training cross-domain text transduction models?
• RQ3 How does the proposed model compare to state-of-the-art unsupervised style transfer and unsupervised machine translation baselines across sentiment, formality, author imitation, decipherment, and related language translation?
• RQ4 What is the impact of parameter sharing, gradient propagation strategies, and entropy regularization on model performance?
• RQ5 Is the unified probabilistic view able to recover competitive results on standard unsupervised MT benchmarks?

Key findings

• The proposed method yields higher reference-BLEU than strong non-generative baselines across sentiment, formality, author imitation, and decipherment tasks.
• On unsupervised MT benchmarks, the approach matches or surpasses current state-of-the-art non-generative systems.
• An entropy-based regularizer in the KL term improves performance by preventing degenerate, over-confident transductions.
• Parameter sharing between the two transfer directions and a shared encoder improves stability and results; removing sharing severely degrades outputs.
• Greedy gradient estimation for latent variables provides a favorable bias-variance trade-off, outperforming more complex gradient estimators in this setting.