[Paper Review] Improving GAN Training via Binarized Representation Entropy (BRE) Regularization

Yanshuai Cao, Gavin Weiguang Ding+2 more|arXiv (Cornell University)|May 9, 2018

Generative Adversarial Networks and Image Synthesis23 references9 citations

TL;DR

This paper introduces Binarized Representation Entropy (BRE) regularization to improve GAN training by encouraging the discriminator's internal hidden layers to produce diverse, high-entropy binary activation patterns, leading to more informative and stable gradients for the generator. The method enhances training stability, convergence speed, and sample quality on both synthetic and real datasets, with improvements in semi-supervised learning accuracy.

ABSTRACT

We propose a novel regularizer to improve the training of Generative Adversarial Networks (GANs). The motivation is that when the discriminator D spreads out its model capacity in the right way, the learning signals given to the generator G are more informative and diverse. These in turn help G to explore better and discover the real data manifold while avoiding large unstable jumps due to the erroneous extrapolation made by D. Our regularizer guides the rectifier discriminator D to better allocate its model capacity, by encouraging the binary activation patterns on selected internal layers of D to have a high joint entropy. Experimental results on both synthetic data and real datasets demonstrate improvements in stability and convergence speed of the GAN training, as well as higher sample quality. The approach also leads to higher classification accuracies in semi-supervised learning.

Motivation & Objective

• To address mode collapse and unstable training in GANs by improving how the discriminator allocates its model capacity.
• To ensure the discriminator provides diverse and informative gradients to the generator by encouraging high-entropy activation patterns in its hidden layers.
• To stabilize GAN training and improve sample quality through a novel regularizer that operates on binary activation patterns of rectifier networks.
• To enhance semi-supervised learning performance by leveraging the improved discriminator capacity allocation from BRE regularization.

Proposed method

• The method introduces a regularizer that maximizes the joint entropy of binary activation patterns across selected hidden layers of the discriminator.
• It leverages the theoretical link between rectifier network capacity and the number of linear regions defined by activation patterns, using entropy as a proxy for capacity diversity.
• The regularizer is applied during training to encourage the discriminator to spread its capacity across more distinct input regions, avoiding degenerate, low-entropy activation patterns.
• The approach is compatible with standard GAN training and does not require architectural changes, only adding a small entropy-based loss term.
• The method is applied to both fake and real data, and also to interpolation points between real and fake samples to further improve generalization.
• The regularization is implemented using a differentiable approximation of entropy on binarized activations, enabling end-to-end training.

Experimental results

Research questions

• RQ1 Can improving the discriminator's capacity allocation through activation pattern diversity lead to more stable and faster GAN training?
• RQ2 Does encouraging high-entropy binary activation patterns in the discriminator improve the quality and diversity of generated samples?
• RQ3 Can BRE regularization enhance generalization in semi-supervised learning by improving the discriminator's representational capacity?
• RQ4 Why do gradient-based diversity regularizers fail in rectifier networks, and how does BRE overcome this limitation?

Key findings

• BRE regularization led to faster convergence and improved training stability, especially in long training runs on SVHN, where it reduced divergence in 100% of trials.
• On CIFAR-10 with feature matching GANs, BRE reduced test error rate to 13.93% (10-ensemble), compared to 14.25% without BRE, indicating better generalization.
• The Inception score improved during training with BRE, showing higher-quality and more diverse samples, particularly in the early training phase.
• The BRE regularizer successfully guided the discriminator to allocate capacity across more linear regions, even when applied only to fake and interpolated samples, not real data.
• The method improved semi-supervised learning performance without architectural changes, demonstrating compatibility and effectiveness across different setups.
• Experiments showed that gradient-based diversity regularizers fail in rectifier networks due to co-linearity of gradients in shared activation patterns, a limitation BRE overcomes by operating on full activation patterns.