[Paper Review] Conditional Variational Autoencoder for Neural Machine Translation

Artidoro Pagnoni, Kevin Liu+1 more|arXiv (Cornell University)|Dec 11, 2018

Natural Language Processing Techniques12 references42 citations

TL;DR

The paper augments NMT with a conditional variational autoencoder using a co-attention based inference network, showing improved BLEU over a discriminative baseline and a VNMT baseline, and investigates posterior collapse mitigation and latent space behavior.

ABSTRACT

We explore the performance of latent variable models for conditional text generation in the context of neural machine translation (NMT). Similar to Zhang et al., we augment the encoder-decoder NMT paradigm by introducing a continuous latent variable to model features of the translation process. We extend this model with a co-attention mechanism motivated by Parikh et al. in the inference network. Compared to the vision domain, latent variable models for text face additional challenges due to the discrete nature of language, namely posterior collapse. We experiment with different approaches to mitigate this issue. We show that our conditional variational model improves upon both discriminative attention-based translation and the variational baseline presented in Zhang et al. Finally, we present some exploration of the learned latent space to illustrate what the latent variable is capable of capturing. This is the first reported conditional variational model for text that meaningfully utilizes the latent variable without weakening the translation model.

Motivation & Objective

• Motivate latent-variable models to capture holistic translation features beyond attention-based encoders.
• Extend CVAE for conditional text generation to NMT with a co-attention inference network.
• Evaluate strategies to mitigate posterior collapse in text VAEs and assess latent space quality.

Proposed method

• Use an encoder-decoder NMT with shared embeddings and RNNs.
• Introduce a neural prior p(z|x) and a neural posterior q(z|x,y) with a co-attention based interaction between source and target.
• Decode conditioned on x, z, and attention-driven context within a Bahdanau-style decoder.
• Train with ELBO objective and KL warm-up (α) to manage posterior collapse.
• Compare vanilla Seq2Seq with attention, VNMT, and CVAE with co-attention on IWSLT 2016 German-English data.

Experimental results

Research questions

• RQ1 Does a CVAE with co-attention improve translation quality over attention-based baselines and VNMT in NMT?
• RQ2 How does co-attention affect the expressiveness of the posterior distribution q(z|x,y)?
• RQ3 What strategies best mitigate posterior collapse in conditional text VAEs without sacrificing translation quality?
• RQ4 What does the learned latent space capture about translation style and variability?

Key findings

• CVAE with co-attention improves BLEU over both vanilla Seq2Seq with attention and the VNMT baseline on German–English translation.
• The co-attention posterior yields lower reconstruction error than VNMT when KL is not zero, indicating a more expressive approximate posterior.
• KL warm-up and KL coefficient strategies help mitigate posterior collapse while maintaining or improving BLEU scores.
• Latent space exploration shows diverse but coherent translations and smooth interpolations, indicating meaningful latent representations.
• Explicitly conditioning the posterior on source–target interactions via co-attention enhances translation-specific latent information.