[Paper Review] Flow Contrastive Estimation of Energy-Based Models

Ruiqi Gao, Erik Nijkamp+4 more|arXiv (Cornell University)|Dec 2, 2019

Generative Adversarial Networks and Image Synthesis71 references4 citations

TL;DR

This paper proposes Flow Contrastive Estimation (FCE), a joint training method that combines energy-based models (EBMs) and normalizing flows to improve generative modeling and semi-supervised learning. By using the flow model as a dynamic noise distribution in noise contrastive estimation and jointly optimizing both models via a shared adversarial objective, FCE enhances sample quality and achieves state-of-the-art performance on semi-supervised image classification, with a 3.87% error rate on SVHN using only 1,000 labeled examples when combined with VAT.

ABSTRACT

This paper studies a training method to jointly estimate an energy-based model and a flow-based model, in which the two models are iteratively updated based on a shared adversarial value function. This joint training method has the following traits. (1) The update of the energy-based model is based on noise contrastive estimation, with the flow model serving as a strong noise distribution. (2) The update of the flow model approximately minimizes the Jensen-Shannon divergence between the flow model and the data distribution. (3) Unlike generative adversarial networks (GAN) which estimates an implicit probability distribution defined by a generator model, our method estimates two explicit probabilistic distributions on the data. Using the proposed method we demonstrate a significant improvement on the synthesis quality of the flow model, and show the effectiveness of unsupervised feature learning by the learned energy-based model. Furthermore, the proposed training method can be easily adapted to semi-supervised learning. We achieve competitive results to the state-of-the-art semi-supervised learning methods.

Motivation & Objective

• To address the limitations of flow models, which assume tractable, invertible transformations and may poorly fit complex data distributions.
• To improve energy-based models (EBMs), which suffer from intractable normalization and expensive sampling, by using a flow model as a strong, adaptive noise distribution.
• To develop a joint training framework that combines the representational flexibility of EBMs with the computational efficiency of flow models.
• To demonstrate improved generative quality and competitive performance in semi-supervised learning using the proposed method.

Proposed method

• The method uses noise contrastive estimation (NCE) to train the energy-based model, where the flow model serves as the contrast distribution, replacing a fixed noise distribution.
• The flow model is updated by minimizing the same NCE objective, making it progressively harder for the EBM to distinguish real data from generated samples, thus improving its quality.
• The joint optimization uses a shared adversarial value function, where both models are updated iteratively to minimize the Jensen-Shannon divergence between the flow model and the data distribution.
• The framework is adapted to semi-supervised learning by incorporating a label consistency objective and, optionally, virtual adversarial training (VAT) for improved robustness.
• The EBM is trained to learn a mixture of class-conditional distributions, with the joint objective encouraging mode separation and smooth clustering within each class.
• The method starts with a pre-trained Glow flow model, and the EBM is first trained in isolation for 50k iterations before joint training begins.

Experimental results

Research questions

• RQ1 Can a flow model serve as a strong, adaptive contrast distribution for training energy-based models via noise contrastive estimation?
• RQ2 How does joint optimization of EBM and flow model improve sample quality compared to training them independently?
• RQ3 Can the proposed method achieve competitive performance in semi-supervised learning without relying on implicit generative models like GANs?
• RQ4 Does the joint training scheme lead to better representation learning and generalization than standard EBM or flow model training?
• RQ5 Can the method be effectively combined with existing regularization techniques such as virtual adversarial training?

Key findings

• FCE achieves a test error rate of 3.87% on the SVHN dataset with only 1,000 labeled examples when combined with virtual adversarial training (VAT), outperforming many state-of-the-art methods.
• The FCE method improves sample quality of the flow model significantly, as shown by qualitative results on both synthetic and real image data.
• The joint training process successfully learns a well-separated, mode-aware energy-based model that captures class-conditional distributions with smooth, connected clusters.
• The FCE-init baseline (EBM trained before joint optimization) already achieves 8.50% error on SVHN with 1,000 labeled examples, showing strong initial performance.
• Adding VAT to FCE further reduces error to 4.47% with 500 labeled examples and 3.87% with 1,000, indicating complementary gains with existing regularization techniques.
• The method demonstrates that combining explicit probabilistic models (EBM and flow) via adversarial contrastive learning leads to improved generalization and representation learning.