[Paper Review] Generalized Energy Based Models

Michael Arbel, Zhou Liang+1 more|arXiv (Cornell University)|Mar 10, 2020

Generative Adversarial Networks and Image Synthesis87 references4 citations

TL;DR

This paper introduces Generalized Energy-Based Models (GEBMs), which combine an implicit base distribution (e.g., a generator) with a learnable energy function to refine probability mass on the data-support manifold. By alternating between training the energy (via a generalized likelihood loss called KALE) and the base, GEBMs achieve higher-quality samples than GANs of equivalent complexity, especially when the data lies on a low-dimensional manifold.

ABSTRACT

We introduce the Generalized Energy Based Model (GEBM) for generative modelling. These models combine two trained components: a base distribution (generally an implicit model), which can learn the support of data with low intrinsic dimension in a high dimensional space; and an energy function, to refine the probability mass on the learned support. Both the energy function and base jointly constitute the final model, unlike GANs, which retain only the base distribution (the "generator"). GEBMs are trained by alternating between learning the energy and the base. We show that both training stages are well-defined: the energy is learned by maximising a generalized likelihood, and the resulting energy-based loss provides informative gradients for learning the base. Samples from the posterior on the latent space of the trained model can be obtained via MCMC, thus finding regions in this space that produce better quality samples. Empirically, the GEBM samples on image-generation tasks are of much better quality than those from the learned generator alone, indicating that all else being equal, the GEBM will outperform a GAN of the same complexity. When using normalizing flows as base measures, GEBMs succeed on density modelling tasks, returning comparable performance to direct maximum likelihood of the same networks.

Motivation & Objective

• Address the limitation of GANs in refining probability mass on low-dimensional data manifolds despite learning the support.
• Overcome the intractability of normalizing constants in energy-based models by introducing a tractable, amortized likelihood estimation method.
• Enable better sampling quality by incorporating the energy function into posterior inference over the latent space of the base generator.
• Demonstrate that GEBMs outperform standard GANs even when the GAN uses a more complex generator, especially in low-dimensional data settings.

Proposed method

• Propose a two-component model: a base distribution (implicit, e.g., a generator) that learns the data support, and an energy function that refines probability mass on that support.
• Train the energy function by maximizing a generalized likelihood using the KALE (KL Approximate Lower-bound Estimate) loss, derived from Donsker-Varadhan and Fenchel duality.
• Use the KALE loss as a training signal for the base generator, ensuring well-defined gradients and weak convergence guarantees.
• Perform alternating optimization: update the energy for a fixed number of steps, then update the base generator using the current energy.
• Sample from the GEBM by performing MCMC in the latent space of the base, using the learned energy to guide the posterior distribution toward high-quality samples.
• Leverage gradient-based MCMC samplers (e.g., based on Eberle et al., 2017) for fast convergence in latent space.

Experimental results

Research questions

• RQ1 Can a model that combines an implicit base with a learnable energy function outperform standard GANs in sample quality while maintaining the same model complexity?
• RQ2 Does the KALE loss provide a stable and informative training signal for both the energy and base components in a jointly trained system?
• RQ3 How does the energy function improve sampling quality by guiding the posterior over the latent space of the base generator?
• RQ4 To what extent does model complexity of the base generator affect the performance gap between GANs and GEBMs?
• RQ5 Can GEBMs achieve competitive density estimation performance when the base is a normalizing flow, matching direct maximum likelihood training?

Key findings

• GEBMs consistently generate higher-quality samples than GANs of the same complexity, even when the GAN uses a more complex generator.
• The performance gap between GANs and GEBMs decreases as the GAN’s generator becomes more expressive, suggesting the energy function captures residual distributional differences.
• GEBMs with a simpler base generator outperform GANs with more complex generators, indicating the energy function provides a more effective refinement mechanism.
• When using normalizing flows as the base, GEBMs achieve density estimation performance comparable to direct maximum likelihood training on the same networks.
• The KALE loss provides well-defined gradients for the base generator and ensures weak convergence under suitable conditions.
• MCMC sampling in the latent space, guided by the learned energy, successfully identifies regions that produce high-quality samples, improving generation quality.