[论文解读] Maximum-Likelihood Augmented Discrete Generative Adversarial Networks

Tong Che, Yanran Li+5 more|arXiv (Cornell University)|Feb 26, 2017

Generative Adversarial Networks and Image Synthesis参考文献 38被引用 175

一句话总结

MaliGAN 提出一种方差降低、最大似然增强的目标函数，用于在离散序列上训练 GAN，从而在文本及其他离散数据生成任务中提高稳定性和性能。

ABSTRACT

Despite the successes in capturing continuous distributions, the application of generative adversarial networks (GANs) to discrete settings, like natural language tasks, is rather restricted. The fundamental reason is the difficulty of back-propagation through discrete random variables combined with the inherent instability of the GAN training objective. To address these problems, we propose Maximum-Likelihood Augmented Discrete Generative Adversarial Networks. Instead of directly optimizing the GAN objective, we derive a novel and low-variance objective using the discriminator's output that follows corresponds to the log-likelihood. Compared with the original, the new objective is proved to be consistent in theory and beneficial in practice. The experimental results on various discrete datasets demonstrate the effectiveness of the proposed approach.

研究动机与目标

• 解决自回归离散生成模型中的曝光偏差和训练不稳定性。
• 引入一个最大似然增强目标，利用具有低方差的判别器输出。
• 通过使用固定目标分布和重要性采样，稳定离散数据的 GAN 训练。
• 在离散 MNIST、诗歌生成和句子级语言建模上展示性能提升。

提出的方法

• 定义一个固定的增强目标分布 q(x) = (D(x)/(1-D(x))) p'(x) ，并以延迟生成器 p' 作为稳定参考。
• Optimize KL(q(x) || p_theta(x)) via an importance-sampling gradient \nabla L_G(theta) ≈ E_p'[ (r_D(x)/Z) ∇_theta log p_theta(x) ], where r_D(x) = D(x)/(1-D(x)).
• 使用基线 b 在梯度估计中降低方差，并在小批量内对权重进行归一化。
• Incorporate variance reduction techniques such as Monte Carlo Tree Search to weight different steps of long sequences.
• Apply mixed MLE-Mali training to combine supervised likelihood with the MaliGAN objective for long sequences.

实验结果

研究问题

• RQ1 Can a maximum-likelihood–based objective, grounded in the discriminator output, stabilize training for discrete sequence GANs?
• RQ2 What theoretical guarantees exist for the MaliGAN objective when the discriminator is optimal or near-optimal?
• RQ3 Do variance reduction techniques (e.g., MCTS, mixed MLE-Mali training) improve stability and performance in discrete sequence generation?
• RQ4 How does MaliGAN perform on discrete MNIST, poetry generation, and sentence-level language modeling compared to MLE and SeqGAN?
• RQ5 Does the proposed approach mitigate exposure bias and loss-evaluation mismatch in practice?

主要发现

• MaliGAN yields a theoretically sound objective that approximates KL(q||p_theta) with a fixed target distribution, improving stability.
• The gradient estimator based on importance sampling with r_D(x) demonstrates lower variance than direct RL rewards from D or log D.
• MaliGAN with variance reduction achieves stable training and superior qualitative and quantitative results on discrete MNIST, poetry generation, and Penn Treebank perplexity tasks.
• Sequential MaliGAN with mixed MLE-Mali training further reduces variance and enhances stability for long sequences.
• MaliGAN shows reduced perplexity and BLEU-2 gains in poetry generation, and lower sentence-level perplexities on PTB compared to MLE and SeqGAN.