[Paper Review] Maximum Entropy Generators for Energy-Based Models

Rithesh Kumar, Ozair, Sherjil+3 more|arXiv (Cornell University)|Jan 24, 2019

Anomaly Detection Techniques and ApplicationsComputer Science46 references62 citations

TL;DR

MEG trains both an energy function and an amortized neural generator to approximate the log-likelihood gradient, using entropy maximization of the generator’s output and a gradient penalty for stability. It yields sharp image samples, supports complete mode coverage, and demonstrates competitive anomaly detection performance.

ABSTRACT

Maximum likelihood estimation of energy-based models is a challenging problem due to the intractability of the log-likelihood gradient. In this work, we propose learning both the energy function and an amortized approximate sampling mechanism using a neural generator network, which provides an efficient approximation of the log-likelihood gradient. The resulting objective requires maximizing entropy of the generated samples, which we perform using recently proposed nonparametric mutual information estimators. Finally, to stabilize the resulting adversarial game, we use a zero-centered gradient penalty derived as a necessary condition from the score matching literature. The proposed technique can generate sharp images with Inception and FID scores competitive with recent GAN techniques, does not suffer from mode collapse, and is competitive with state-of-the-art anomaly detection techniques.

Motivation & Objective

• Motivate and address the difficulty of training energy-based models (EBMs) due to intractable log-likelihood gradients.
• Propose a joint framework to learn an energy function and an amortized sampler that approximates p_theta.
• Maximize the entropy of the generator’s outputs to align p_G with p_theta and stabilize training.
• Leverage nonparametric mutual information estimators to maximize generator entropy.
• Show that MEG improves image sharpness, mode coverage, and anomaly detection compared to baselines.

Proposed method

• Replace the model distribution p_theta with a neural generator G(w) producing samples G(z) from a latent prior z~p_z.
• Minimize KL(p_G||p_theta) which leads to a generator loss combining generator entropy and energy evaluation: L_G = -I_JSD(G(Z),Z) + E_theta[E(G(Z))].
• Train the energy function E_theta by gradient: L_E = E_{x~p_D}[E_theta(x)] - E_{z~p_z}[E_theta(G(z))].
• Maximize generator entropy via nonparametric mutual information estimation, using I_JSD(G(Z),Z) as in Belghazi et al. (2018) and related work.
• Stabilize training with a zero-centered gradient penalty inspired by score matching to keep samples near energy function critical points.
• Optionally perform latent-space MCMC (MALA with Metropolis-Hastings step) to bias sampling toward high-density regions in the latent space.

Experimental results

Research questions

• RQ1 Can an amortized neural sampler approximate the energy-based model's negative phase gradient without relying on costly MCMC in data space?
• RQ2 Does maximizing the entropy of the generator output help prevent mode collapse and ensure coverage of all data modes?
• RQ3 Can the MEG framework produce competitive image samples and preserve sharpness without sacrificing mode diversity?
• RQ4 Is the learned energy function useful for anomaly detection on standard benchmarks?

Key findings

• MEG generates CIFAR-10 samples competitive with WGAN-GP according to Inception Score and Fréchet Inception Distance.
• MEG accurately captures all modes on 4-StackedMNIST, with a lower KL divergence than several baselines.
• Latent-space MALA sampling improves perceptual sample quality compared to visible-space MCMC.
• MEG achieves strong anomaly detection performance on KDD99, rivaling state-of-the-art methods.
• MEG yields sharp, less blurry samples relative to typical maximum likelihood EBMs and avoids mode collapse.