[Paper Review] Guiding Energy-based Models via Contrastive Latent Variables

Hankook Lee, Jongheon Jeong+2 more|arXiv (Cornell University)|Mar 6, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Contrastive Latent-guided Energy Learning (CLEL), a framework that improves energy-based models (EBMs) by using contrastive representation learning (CRL) as a latent variable to guide training. By jointly training a spherical latent-variable EBM with a contrastive encoder, CLEL achieves state-of-the-art FID scores with faster, more memory-efficient training than prior EBM methods, including those using VAEs or diffusion models.

ABSTRACT

An energy-based model (EBM) is a popular generative framework that offers both explicit density and architectural flexibility, but training them is difficult since it is often unstable and time-consuming. In recent years, various training techniques have been developed, e.g., better divergence measures or stabilization in MCMC sampling, but there often exists a large gap between EBMs and other generative frameworks like GANs in terms of generation quality. In this paper, we propose a novel and effective framework for improving EBMs via contrastive representation learning (CRL). To be specific, we consider representations learned by contrastive methods as the true underlying latent variable. This contrastive latent variable could guide EBMs to understand the data structure better, so it can improve and accelerate EBM training significantly. To enable the joint training of EBM and CRL, we also design a new class of latent-variable EBMs for learning the joint density of data and the contrastive latent variable. Our experimental results demonstrate that our scheme achieves lower FID scores, compared to prior-art EBM methods (e.g., additionally using variational autoencoders or diffusion techniques), even with significantly faster and more memory-efficient training. We also show conditional and compositional generation abilities of our latent-variable EBMs as their additional benefits, even without explicit conditional training. The code is available at https://github.com/hankook/CLEL.

Motivation & Objective

• Address the long-standing challenge of training instability and slow convergence in energy-based models (EBMs).
• Improve EBM generation quality to close the gap with state-of-the-art generative models like GANs and diffusion models.
• Leverage low-cost, self-supervised representations from contrastive learning (CRL) to guide EBM training without requiring additional generative models.
• Enable efficient joint training of EBMs and contrastive encoders while preserving the explicit density property of EBMs.
• Demonstrate that contrastive representations can serve as effective latent variables for structured data modeling in EBMs.

Proposed method

• Introduce a spherical latent-variable EBM that models the joint distribution $ p_{\theta}(\mathbf{x}, \mathbf{z}) $, where $ \mathbf{z} $ is a contrastive representation.
• Train a contrastive encoder $ h_{\phi} $ using instance-level contrastive learning (e.g., SimCLR) to produce semantic, normalized latent codes $ \mathbf{z} \in \mathbb{S}^{d-1} $.
• Use stop-gradient operations to decouple the contrastive learning objective from the EBM training, enabling joint optimization.
• Design a joint training objective that combines EBM training with contrastive learning, using the contrastive loss to guide the EBM toward better data structure understanding.
• Introduce a projection head $ g_{\theta} $ to map the EBM energy output to a latent space compatible with contrastive learning.
• Ensure the latent variable $ \mathbf{z} $ is normalized to unit sphere to align with standard CRL objectives and improve training stability.

Experimental results

Research questions

• RQ1 Can contrastive representation learning serve as an effective latent variable to guide energy-based model training?
• RQ2 Does using contrastive features improve the sample quality and training efficiency of EBMs compared to standard EBM training?
• RQ3 Can the proposed framework achieve competitive FID scores without relying on expensive auxiliary models like VAEs or diffusion models?
• RQ4 How robust is the framework to different contrastive learning methods (e.g., SimCLR, BYOL, MAE) for the encoder?
• RQ5 To what extent can the learned latent-variable EBM perform conditional and compositional generation without explicit conditional training?

Key findings

• CLEL achieves a FID score of 35.73 on CIFAR-10, significantly outperforming the baseline EBM (42.46) and prior SOTA EBM methods.
• The method reduces FID by 16.4% compared to the baseline and achieves better performance than EBM methods enhanced with VAEs or diffusion models.
• The framework is compatible with multiple self-supervised representation learning methods, including BYOL (FID: 36.31) and MAE (FID: 37.67), showing broad applicability.
• The contrastive latent variable enables zero-shot conditional and compositional generation without explicit conditional training, demonstrating disentangled representation learning.
• The model achieves high OOD detection performance (AUC: 0.8723), indicating that the learned EBM captures meaningful data distribution and uncertainty.
• Ablation studies confirm that both the MLP projection head and the negative sampling distribution $ p_{\theta}(\tilde{\mathbf{z}}) $ are critical for performance, with the identity head degrading results significantly (FID: 86.02).