[Paper Review] Energy-based Generative Adversarial Network

Junbo Zhao, Michaël Mathieu+1 more|arXiv (Cornell University)|Sep 11, 2016

Generative Adversarial Networks and Image Synthesis23 references893 citations

TL;DR

EBGAN reinterprets the GAN discriminator as an energy function, enabling auto-encoder based energy measures and improved training stability, with high-resolution image generation demonstrated.

ABSTRACT

We introduce the "Energy-based Generative Adversarial Network" model (EBGAN) which views the discriminator as an energy function that attributes low energies to the regions near the data manifold and higher energies to other regions. Similar to the probabilistic GANs, a generator is seen as being trained to produce contrastive samples with minimal energies, while the discriminator is trained to assign high energies to these generated samples. Viewing the discriminator as an energy function allows to use a wide variety of architectures and loss functionals in addition to the usual binary classifier with logistic output. Among them, we show one instantiation of EBGAN framework as using an auto-encoder architecture, with the energy being the reconstruction error, in place of the discriminator. We show that this form of EBGAN exhibits more stable behavior than regular GANs during training. We also show that a single-scale architecture can be trained to generate high-resolution images.

Motivation & Objective

• Introduce an energy-based formulation of GANs where the discriminator outputs an energy score rather than a probability.
• Show that under this framework, the generator minimizes energy regions outside the data manifold while the discriminator assigns low energy to real data.
• Demonstrate that using an auto-encoder as the discriminator yields stable training and can generate high-resolution images without multi-scale setups.
• Provide theoretical results on equilibrium conditions where the generator matches the data distribution under a simple hinge loss.
• Investigate regularization techniques (like repelling regularizers) to encourage coverage of multiple data modes.

Proposed method

• Define D as an energy function and train with a margin loss that pushes D(x) low for real data and D(G(z)) above a margin m.
• Use a generator loss L_G(z)=D(G(z)) and a discriminator loss L_D(x,z)=D(x)+[m−D(G(z))]^+ where [·]^+ is the hinge.
• Instantiate D as an auto-encoder, with energy equal to the reconstruction error ||Dec(Enc(x))−x||.
• Argue that energy-based framing allows flexible architectures and loss functionals beyond binary classifiers.
• Introduce a repelling regularizer (Pulling-away Term) to promote diverse representations and mitigate mode collapse.
• Discuss regularization of auto-encoders to avoid identity mappings and ensure energy is higher outside the data manifold.

Experimental results

Research questions

• RQ1 Does an energy-based GAN (EBGAN) framework with an auto-encoder discriminator converge to the data distribution under Nash equilibrium?
• RQ2 How does the margin loss influence equilibrium and training stability in EBGANs?
• RQ3 Can repelling regularizers improve mode coverage and diversity of generated samples?
• RQ4 Can high-resolution images be generated with a single-scale EBGAN using auto-encoder discriminators?
• RQ5 What are the impacts of architectural choices on stability and quality in EBGANs compared to traditional GANs?

Key findings

• Under a simple hinge loss, the Nash equilibrium yields p_G = p_data, meaning generated samples match the data distribution.
• Auto-encoder discriminators provide a flexible energy surface and can yield more stable training than standard GAN discriminators.
• EBGAN can generate high-resolution images (256×256) on ImageNet without multi-scale architectures.
• Regularizers like the repelling term improve sample diversity and help cover multiple data modes.
• EBGANs show more reliable training than GANs in MNIST grid searches and enable semi-supervised extensions with Ladder Networks.
• Combining EBGAN with deep convolutional architectures enables realistic generation on LSUN Bedroom and CelebA datasets.