[Paper Review] Latent Alignment and Variational Attention

Yuntian Deng, Yoon Kim+3 more|arXiv (Cornell University)|Jul 10, 2018

Topic ModelingComputer Science85 citations

TL;DR

The paper introduces variational attention as latent alignment models for neural attention, showing variational (categorical) attention can outperform soft and hard attention on NMT and VQA while maintaining training efficiency comparable to soft attention.

ABSTRACT

Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property makes it difficult to compare attention to other alignment approaches, to compose it with probabilistic models, and to perform posterior inference conditioned on observed data. A related latent approach, hard attention, fixes these issues, but is generally harder to train and less accurate. This work considers variational attention networks, alternatives to soft and hard attention for learning latent variable alignment models, with tighter approximation bounds based on amortized variational inference. We further propose methods for reducing the variance of gradients to make these approaches computationally feasible. Experiments show that for machine translation and visual question answering, inefficient exact latent variable models outperform standard neural attention, but these gains go away when using hard attention based training. On the other hand, variational attention retains most of the performance gain but with training speed comparable to neural attention.

Motivation & Objective

• Motivate latent variable alignment in attention beyond soft attention.
• Develop variational attention architectures (categorical and relaxed) with amortized inference.
• Tighten bounds on marginal likelihood to improve alignment estimates.
• Empirically compare variational attention to soft and hard attention on NMT and VQA.
• Investigate training and inference strategies to enable scalable latent alignment models.

Proposed method

• Formalize latent alignment as z drawn from an alignment distribution a(x, tilde{x}; theta).
• Introduce two variational attention variants: categorical (amortized VI with REINFORCE baseline) and relaxed (Dirichlet) attention.
• Use amortized inference networks enc to parameterize q(z; lambda) and optimize the ELBO with KL regularization.
• Train with objective tied to log p(y|x, tilde{x}) via variational bounds, improving gradient estimators (baseline, REINFORCE, or reparameterization).
• Apply to Neural Machine Translation (NMT) and Visual Question Answering (VQA) with domain-specific encoders and attention modules.

Experimental results

Research questions

• RQ1 Does marginalizing latent alignments via variational inference improve over soft and hard attention in practice?
• RQ2 Can variational attention provide better posterior alignments and interpretability without prohibitive training cost?
• RQ3 How do categorical vs relaxed (Dirichlet) variational alignments compare in NMT and VQA?
• RQ4 What training/inference strategies (baselines, REINFORCE, Gumbel-Softmax, RWS) best stabilize variational attention?
• RQ5 Is exact marginal likelihood advantageous when feasible, and how close can variational methods get to it?

Key findings

• Exact marginal likelihood can improve over soft attention when feasible.
• Variational (categorical) attention surpasses both soft and hard attention in NMT and VQA in log-likelihood metrics, with training efficiency close to soft attention.
• Hard attention underperforms soft attention, indicating Jensen’s inequality is a poor bound in practice.
• In NMT, variational attention with enumeration or sampling achieves comparable or better BLEU and PPL than soft and hard baselines on standard benchmarks, including WMT13/17 scale.
• Relaxed (Dirichlet) attention underperforms relative to categorical variants in their experiments, while providing posterior inference capabilities.