[论文解读] On the Effects of Batch and Weight Normalization in Generative Adversarial Networks

Sitao Xiang, Hao Li|arXiv (Cornell University)|Apr 13, 2017

Generative Adversarial Networks and Image Synthesis参考文献 14被引用 70

一句话总结

本文评估批量归一化与为GANs提出的权重归一化，提出一个基于ReLU的权重归一化方法的翻译版本，并显示在类似DCGAN的架构中，WN在稳定性和重建质量方面优于BN。

ABSTRACT

Generative adversarial networks (GANs) are highly effective unsupervised learning frameworks that can generate very sharp data, even for data such as images with complex, highly multimodal distributions. However GANs are known to be very hard to train, suffering from problems such as mode collapse and disturbing visual artifacts. Batch normalization (BN) techniques have been introduced to address the training. Though BN accelerates the training in the beginning, our experiments show that the use of BN can be unstable and negatively impact the quality of the trained model. The evaluation of BN and numerous other recent schemes for improving GAN training is hindered by the lack of an effective objective quality measure for GAN models. To address these issues, we first introduce a weight normalization (WN) approach for GAN training that significantly improves the stability, efficiency and the quality of the generated samples. To allow a methodical evaluation, we introduce squared Euclidean reconstruction error on a test set as a new objective measure, to assess training performance in terms of speed, stability, and quality of generated samples. Our experiments with a standard DCGAN architecture on commonly used datasets (CelebA, LSUN bedroom, and CIFAR-10) indicate that training using WN is generally superior to BN for GANs, achieving 10% lower mean squared loss for reconstruction and significantly better qualitative results than BN. We further demonstrate the stability of WN on a 21-layer ResNet trained with the CelebA data set. The code for this paper is available at https://github.com/stormraiser/gan-weightnorm-resnet

研究动机与目标

• 评估批量归一化对GAN训练稳定性与样本质量的影响。
• 提出基于权重归一化的GAN训练变体并评估其收益。
• 引入一个以重建为导向的客观指标，用于评估GAN训练速度、稳定性和质量。
• 在DCGAN架构上，在CelebA、LSUN、CIFAR-10数据集上演示。

提出的方法

• 引入一种改进的权重归一化（WN）技术用于GAN训练，并与批量归一化（BN）进行比较。
• 在权重归一化层中定义一个翻译的ReLU（TReLU），以在不引入大幅均值偏移的情况下保持表示能力。
• 提出在测试集上使用目标重建损失，利用梯度下降找到使 ||G(z)-x||^2 最小的潜在代码。
• 使用RMSProp进行优化，并在CelebA、LSUN卧室以及CIFAR-10上进行类似DCGAN的架构实验，还包括一个21层ResNet实验。
• 通过最终重建损失评估重建质量，通过迭代中生成样本的演变来评估训练稳定性。

实验结果

研究问题

• RQ1 批量归一化在标准基准上是改善还是降低GAN训练稳定性和样本质量？
• RQ2 就稳定性和重建质量而言，权重归一化的形式是否优于BN？
• RQ3 基于重建的目标是否能可靠监控GAN训练进展并反映定性样本质量？
• RQ4 提议的WN方法在更深的架构（如ResNet）和多数据集上表现如何？

主要发现

• 权重归一化比未归一化模型的最终重建损失约低10.5%。
• 权重归一化比批量归一化提供更稳定的训练和更好的重建质量。
• 批量归一化可能引发不稳定和伪影，而原生GAN在某些数据集上可能崩溃。
• 采用权重归一化训练的21层ResNet GAN保持稳定并产生多样化、高质量的样本。
• BN在早期训练中加速但相较于WN可能降低长期稳定性和样本多样性。
• WN提供比BN更快更稳定的训练，且样本质量等于或高于非归一化模型。