[Paper Review] Neural Sampling in Hierarchical Exponential-family Energy-based Models

Xingsi Dong, Si Wu|arXiv (Cornell University)|Oct 12, 2023

Neural Networks and ApplicationsComputer Science3 citations

TL;DR

This paper proposes the Hierarchical Exponential-family Energy-based (HEE) model, which enables localized, biologically plausible inference and learning by decomposing the partition function across layers and using fast-dynamics neurons to sample the normalization term. The model achieves efficient, convergent training without a negative phase and matches biological neural dynamics, including oscillations and transient responses, while generating high-quality images comparable to state-of-the-art EBMs.

ABSTRACT

Bayesian brain theory suggests that the brain employs generative models to understand the external world. The sampling-based perspective posits that the brain infers the posterior distribution through samples of stochastic neuronal responses. Additionally, the brain continually updates its generative model to approach the true distribution of the external world. In this study, we introduce the Hierarchical Exponential-family Energy-based (HEE) model, which captures the dynamics of inference and learning. In the HEE model, we decompose the partition function into individual layers and leverage a group of neurons with shorter time constants to sample the gradient of the decomposed normalization term. This allows our model to estimate the partition function and perform inference simultaneously, circumventing the negative phase encountered in conventional energy-based models (EBMs). As a result, the learning process is localized both in time and space, and the model is easy to converge. To match the brain's rapid computation, we demonstrate that neural adaptation can serve as a momentum term, significantly accelerating the inference process. On natural image datasets, our model exhibits representations akin to those observed in the biological visual system. Furthermore, for the machine learning community, our model can generate observations through joint or marginal generation. We show that marginal generation outperforms joint generation and achieves performance on par with other EBMs.

Motivation & Objective

• To develop a generative model that supports both sampling-based Bayesian inference and localized learning in time and space, consistent with brain function.
• To address the challenge of partition function estimation in energy-based models (EBMs), which traditionally requires a non-local negative phase.
• To integrate neural adaptation into inference dynamics to accelerate convergence and mimic biological neural phenomena.
• To validate the model’s ability to generate realistic data and produce representations resembling those in the biological visual system.

Proposed method

• Proposes the Hierarchical Exponential-family Energy-based (HEE) model, which decomposes the global partition function into a sum of layer-wise normalization terms, reducing the required sample space from a product to a sum.
• Uses a group of fast-dynamics neurons with shorter time constants to sample the gradient of the decomposed log-partition function, enabling local estimation of the normalization term.
• Introduces neural adaptation as a momentum-like term in the inference process, transforming the dynamics into a second-order Langevin process that accelerates convergence.
• Employs marginal generation via latent space MCMC, which outperforms joint generation and matches performance of other EBMs on image generation tasks.
• Incorporates biological constraints such as receptive fields during training on CIFAR10, aligning the model’s representations with those observed in the visual cortex.
• Utilizes a hierarchical Markov structure similar to diffusion models, but unfolds the Markov chain across neural layers rather than over time.

Experimental results

Research questions

• RQ1 Can a hierarchical energy-based model enable both efficient inference and localized learning in time and space, consistent with neural computation in the brain?
• RQ2 How can the partition function estimation in EBMs be localized to avoid the non-local negative phase?
• RQ3 Can intrinsic neural adaptation serve as a momentum mechanism to accelerate sampling in non-convex energy landscapes?
• RQ4 Does the HEE model generate representations of semantic features (e.g., orientation, color, category) that resemble those in the biological visual system?
• RQ5 Can marginal generation in the HEE model achieve performance comparable to other state-of-the-art energy-based models?

Key findings

• The HEE model eliminates the need for a negative phase by decomposing the partition function across layers, enabling fully localized learning in both time and space.
• The use of fast-dynamics neurons to sample the layer-wise normalization term allows the model to estimate the partition function and perform inference simultaneously.
• Incorporating neural adaptation as a momentum term accelerates inference and induces neural phenomena such as oscillations and transients, commonly observed in biological systems.
• On FashionMNIST and CIFAR10, the HEE model achieves image generation performance on par with other state-of-the-art energy-based models.
• The model's learned representations for semantic features such as orientation, color, and category closely resemble those found in the biological visual system.
• Marginal generation in the HEE model outperforms joint generation and matches the performance of other EBMs, demonstrating its strong generative capacity.