[Paper Review] Maximum-Likelihood Augmented Discrete Generative Adversarial Networks

Tong Che, Yanran Li+5 more|arXiv (Cornell University)|Feb 26, 2017

Generative Adversarial Networks and Image SynthesisComputer Science38 references175 citations

TL;DR

MaliGAN proposes a variance-reduced, maximum-likelihood augmented objective for training GANs on discrete sequences, improving stability and performance in text and other discrete data generation tasks.

ABSTRACT

Despite the successes in capturing continuous distributions, the application of generative adversarial networks (GANs) to discrete settings, like natural language tasks, is rather restricted. The fundamental reason is the difficulty of back-propagation through discrete random variables combined with the inherent instability of the GAN training objective. To address these problems, we propose Maximum-Likelihood Augmented Discrete Generative Adversarial Networks. Instead of directly optimizing the GAN objective, we derive a novel and low-variance objective using the discriminator's output that follows corresponds to the log-likelihood. Compared with the original, the new objective is proved to be consistent in theory and beneficial in practice. The experimental results on various discrete datasets demonstrate the effectiveness of the proposed approach.

Motivation & Objective

• Address exposure bias and training instability in autoregressive discrete generative models.
• Introduce a maximum-likelihood augmented objective that leverages discriminator outputs with low variance.
• Stabilize training of GANs for discrete data by using a fixed target distribution and importance sampling.
• Demonstrate improved performance on discrete MNIST, poem generation, and sentence-level language modeling.

Proposed method

• Define a fixed augmented target distribution q(x) = (D(x)/(1-D(x))) p'(x) with a delayed generator p' as a stabilizing reference.
• Optimize KL(q(x) || p_theta(x)) via an importance-sampling gradient abla L_G(theta) ≈ E_p'[ (r_D(x)/Z) ∇_theta log p_theta(x) ], where r_D(x) = D(x)/(1-D(x)).
• Use a baseline b to reduce variance in the gradient estimator and normalize weights within mini-batches.
• Incorporate variance reduction techniques such as Monte Carlo Tree Search to weight different steps of long sequences.
• Apply mixed MLE-Mali training to combine supervised likelihood with the MaliGAN objective for long sequences.

Experimental results

Research questions

• RQ1 Can a maximum-likelihood–based objective, grounded in the discriminator output, stabilize training for discrete sequence GANs?
• RQ2 What theoretical guarantees exist for the MaliGAN objective when the discriminator is optimal or near-optimal?
• RQ3 Do variance reduction techniques (e.g., MCTS, mixed MLE-Mali training) improve stability and performance in discrete sequence generation?
• RQ4 How does MaliGAN perform on discrete MNIST, poetry generation, and sentence-level language modeling compared to MLE and SeqGAN?
• RQ5 Does the proposed approach mitigate exposure bias and loss-evaluation mismatch in practice?

Key findings

• MaliGAN yields a theoretically sound objective that approximates KL(q||p_theta) with a fixed target distribution, improving stability.
• The gradient estimator based on importance sampling with r_D(x) demonstrates lower variance than direct RL rewards from D or log D.
• MaliGAN with variance reduction achieves stable training and superior qualitative and quantitative results on discrete MNIST, poetry generation, and Penn Treebank perplexity tasks.
• Sequential MaliGAN with mixed MLE-Mali training further reduces variance and enhances stability for long sequences.
• MaliGAN shows reduced perplexity and BLEU-2 gains in poetry generation, and lower sentence-level perplexities on PTB compared to MLE and SeqGAN.