[论文解读] Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow

Xue Bin Peng, Angjoo Kanazawa+3 more|arXiv (Cornell University)|Oct 1, 2018

Adversarial Robustness in Machine Learning参考文献 40被引用 101

一句话总结

简要：引入变分判别器瓶颈（VDB）通过信息瓶颈正则化判别器，稳定对抗学习，在模仿学习、逆强化学习和GAN中提高性能。

ABSTRACT

Adversarial learning methods have been proposed for a wide range of applications, but the training of adversarial models can be notoriously unstable. Effectively balancing the performance of the generator and discriminator is critical, since a discriminator that achieves very high accuracy will produce relatively uninformative gradients. In this work, we propose a simple and general technique to constrain information flow in the discriminator by means of an information bottleneck. By enforcing a constraint on the mutual information between the observations and the discriminator's internal representation, we can effectively modulate the discriminator's accuracy and maintain useful and informative gradients. We demonstrate that our proposed variational discriminator bottleneck (VDB) leads to significant improvements across three distinct application areas for adversarial learning algorithms. Our primary evaluation studies the applicability of the VDB to imitation learning of dynamic continuous control skills, such as running. We show that our method can learn such skills directly from \emph{raw} video demonstrations, substantially outperforming prior adversarial imitation learning methods. The VDB can also be combined with adversarial inverse reinforcement learning to learn parsimonious reward functions that can be transferred and re-optimized in new settings. Finally, we demonstrate that VDB can train GANs more effectively for image generation, improving upon a number of prior stabilization methods.

研究动机与目标

• 通过在判别器中使用信息瓶颈来约束信息流，从而动机化并稳定对抗学习。
• 开发一种基于变分界界限的实用方法（VDB），调节判别器准确性以维持有信息的梯度。
• 在三个领域展示改进：来自动作捕捉/视频的运动模仿、对抗性IRL，以及基于GAN的图像生成。

提出的方法

• 引入一个编码器E，将输入x映射到随机潜变量z ~ E(z|x)。
• 通过使用先验r(z)的KL基上界，强加互信息约束I(X;Z) ≤ Ic。
• 使用对偶变量β优化一个拉格朗日函数，以自适应地强制信息预算（β通过对偶梯度下降更新）。
• 判别器D对来自E(z|x)的样本z进行分类，而不是直接对x分类；对D(z)使用sigmoid输出二元真/假。
• 使用一个简化的生成器目标，在编码器均值μE(x)处评估D，以近似对z的期望。
• 将此VDB框架应用于GAIL（VAIL）、VAIRL，并扩展到基于像素的视频演示模仿。

实验结果

研究问题

• RQ1 通过在对抗学习中约束信息流（信息瓶颈）是否能改善稳定性与梯度质量？
• RQ2 变分判别瓶颈（VDB）在来自演示（包括视频）的模仿学习和对抗性IRL设置中是否提升性能？
• RQ3 VDB是否对基于GAN的图像生成有益，超越标准稳定化方法？
• RQ4 自适应β优化如何影响训练动态与最终性能？
• RQ5 将VDB应用于VAIRL及相关对抗性IRL公式时会获得哪些提升？

主要发现

• VDB 平滑了判别器的决策边界并提供有信息的梯度，防止梯度消失，提升学习。
• VAIL（将VDB应用于对抗性模仿学习）在多项技能的动作捕捉与视频演示模仿方面表现更好，超越GAIL变体和BC。
• 启用VDB的VAIRL及VAIRL-GP变体在迁移/奖励学习方面与基线及AIRL变体相比具有竞争力或更优。
• 使用对偶梯度下降的自适应β有效执行信息约束，并在基于视频的运动模仿中实现最佳整体性能。
• 在图像生成中，基于VDB的GAN（VGAN）相对于若干先前的稳定化方法提升了稳定性和性能。