[Paper Review] Implicit Generation and Modeling in Energy-Based Models

Yilun Du, Igor Mordatch|arXiv (Cornell University)|Jan 1, 2019

Anomaly Detection Techniques and Applications28 references168 citations

TL;DR

This paper presents scalable MCMC-based training for energy-based models (EBMs) on continuous neural networks, enabling high-quality implicit generation on ImageNet and robotic trajectory data. It achieves state-of-the-art performance in sample quality, out-of-distribution detection, adversarial robustness, and long-horizon trajectory prediction, demonstrating the versatility of EBMs in diverse learning tasks.

ABSTRACT

Energy based models (EBMs) are appealing due to their generality and simplicity in likelihood modeling, but have been traditionally difficult to train. We present techniques to scale MCMC based EBM training on continuous neural networks, and we show its success on the high-dimensional data domains of ImageNet32x32, ImageNet128x128, CIFAR-10, and robotic hand trajectories, achieving better samples than other likelihood models and nearing the performance of contemporary GAN approaches, while covering all modes of the data. We highlight some unique capabilities of implicit generation such as compositionality and corrupt image reconstruction and inpainting. Finally, we show that EBMs are useful models across a wide variety of tasks, achieving state-of-the-art out-of-distribution classification, adversarially robust classification, state-of-the-art continual online class learning, and coherent long term predicted trajectory rollouts.

Motivation & Objective

• Address the long-standing challenge of training energy-based models (EBMs) on high-dimensional continuous data due to computational and sampling difficulties.
• Enable effective implicit generation in EBMs for complex data distributions such as ImageNet and robotic hand trajectories.
• Demonstrate that EBMs can achieve competitive sample quality compared to GANs while covering all data modes.
• Explore the utility of EBMs beyond generation, including out-of-distribution detection, adversarial robustness, and continual learning.
• Investigate the unique capabilities of implicit generation, such as compositionality and inpainting, in a unified EBM framework.

Proposed method

• Adopt MCMC-based training for EBMs on continuous neural networks, enabling scalable optimization in high-dimensional spaces.
• Leverage score-based generative modeling principles to improve sampling efficiency and training stability in EBM training.
• Use a noise-contrastive estimation-like objective to approximate the energy function without explicit likelihood computation.
• Integrate MCMC sampling with neural network parameterization to enable end-to-end training on large-scale datasets.
• Apply the trained EBM for downstream tasks such as out-of-distribution detection and adversarially robust classification.
• Utilize the implicit model structure to perform compositionality and image reconstruction tasks without explicit decoder networks.

Experimental results

Research questions

• RQ1 Can MCMC-based training of energy-based models scale to high-dimensional data like ImageNet and robotic trajectories?
• RQ2 How does the sample quality of EBM-based implicit generation compare to likelihood-based models and GANs?
• RQ3 To what extent can EBMs support compositional generation and image inpainting without explicit generative heads?
• RQ4 Can EBMs achieve state-of-the-art performance in out-of-distribution detection and adversarial robustness?
• RQ5 How effective are EBMs for continual online class learning and long-term trajectory prediction?

Key findings

• The proposed MCMC-based EBM training achieves sample quality on ImageNet32x32 and ImageNet128x128 that rivals or exceeds that of contemporary GANs and other likelihood-based models.
• The model successfully covers all modes of the data distribution, avoiding mode collapse common in generative models.
• EBMs demonstrate strong performance in out-of-distribution detection, achieving state-of-the-art results on benchmark benchmarks.
• The model exhibits adversarial robustness, outperforming standard generative models in adversarially trained classification tasks.
• EBMs enable state-of-the-art continual online class learning, adapting to new classes without catastrophic forgetting.
• The implicit generation framework supports coherent long-term trajectory rollouts and effective image inpainting and reconstruction.