[论文解读] Language GANs Falling Short

M. Caccia, Lucas Caccia+4 more|arXiv (Cornell University)|Nov 6, 2018

Topic Modeling参考文献 38被引用 81

一句话总结

本文表明，在质量与多样性方面，经过良好调优的最大似然语言模型在基于GAN的文本生成中占据主导地位，且使用温度扫描评估框架。

ABSTRACT

Generating high-quality text with sufficient diversity is essential for a wide range of Natural Language Generation (NLG) tasks. Maximum-Likelihood (MLE) models trained with teacher forcing have consistently been reported as weak baselines, where poor performance is attributed to exposure bias (Bengio et al., 2015; Ranzato et al., 2015); at inference time, the model is fed its own prediction instead of a ground-truth token, which can lead to accumulating errors and poor samples. This line of reasoning has led to an outbreak of adversarial based approaches for NLG, on the account that GANs do not suffer from exposure bias. In this work, we make several surprising observations which contradict common beliefs. First, we revisit the canonical evaluation framework for NLG, and point out fundamental flaws with quality-only evaluation: we show that one can outperform such metrics using a simple, well-known temperature parameter to artificially reduce the entropy of the model's conditional distributions. Second, we leverage the control over the quality / diversity trade-off given by this parameter to evaluate models over the whole quality-diversity spectrum and find MLE models constantly outperform the proposed GAN variants over the whole quality-diversity space. Our results have several implications: 1) The impact of exposure bias on sample quality is less severe than previously thought, 2) temperature tuning provides a better quality / diversity trade-off than adversarial training while being easier to train, easier to cross-validate, and less computationally expensive. Code to reproduce the experiments is available at github.com/pclucas14/GansFallingShort

研究动机与目标

• 研究GAN基于文本生成是否在质量和多样性上能够超越MLE基线。
• 评估暴露偏差与不可微GAN训练对样本质量与多样性的影响。
• 提出一个鲁棒、低偏差的评估框架，在质量-多样性范围内比较NLG模型。
• 使用温度受控采样及其他解码策略来量化权衡。

提出的方法

• 定义一个玻尔兹曼温度参数，用于控制自回归生成中的熵。
• 引入温度扫描，以映射各模型之间的质量-多样性权衡。
• 在温度扫描下，将自回归MLE基线与各种GAN变体（RL和非RL）进行比较。
• 使用局部指标（如BLEU、自BLEU等）和全局指标（Language Model分数、Reverse LM分数）进行评估。
• 分析解码策略（温度调优、随机束搜索、生成器拒绝）在质量 vs. 多样性方面的影响。

实验结果

研究问题

• RQ1 受控温度的采样是否为跨NLG模型的质量和多样性提供了公平、偏差较低的比较？
• RQ2 MLE模型是否在整个质量-多样性范围内优于基于GAN的文本生成器？
• RQ3 不同模型的解码策略如何影响感知的质量-多样性权衡？
• RQ4 暴露偏差是文本生成的主要瓶颈吗，还是GAN的优化/训练挑战占主导？
• RQ5 不同评估/探针技术在NLG模型上的实际成本和偏差是什么？

主要发现

模型	NLL oracle
SeqGAN (Yu et al., 2017)	8.74
RankGAN (Lin et al., 2017)	8.25
LeakGAN (Guo et al., 2017)	7.04
IRL (Shi et al., 2018)	6.91
MLE (α=1.0)	9.40
MLE (α=0.4)	5.50
MLE (α=0.001)	4.58

• 在温度扫描下，MLE模型在整个质量-多样性空间中持续优于GAN变体。
• 降低温度提高质量但降低多样性；更高温度增加多样性，但在可控的方式下可能损害质量。
• GAN训练降低生成分布的熵，从而导致多样性降低和权衡较差。
• 像随机束搜索和生成器拒绝等解码方法存在偏差和计算成本，限制了其有效性；温度扫描提供高效、无偏的评估。
• 仅温度调优往往足以揭示性能差异，在合理计算量下，MLE提供最佳的质量-多样性平衡。