[Paper Review] Semiparametric energy-based probabilistic models

Jan Humplik, Gašper Tkačik|arXiv (Cornell University)|May 24, 2016

Neural dynamics and brain function23 references3 citations

TL;DR

This paper introduces semiparametric energy-based models that generalize traditional energy-based models by learning a nonlinear transformation of the energy function via a flexible, nonparametric representation. By modeling the nonlinearity using piecewise-constant basis functions and optimizing it jointly with the energy parameters, the method captures complex probability distributions with large dynamic ranges—demonstrated effectively on retinal neural data, where it outperforms standard models.

ABSTRACT

Probabilistic models can be defined by an energy function, where the probability of each state is proportional to the exponential of the state's negative energy. This paper considers a generalization of energy-based models in which the probability of a state is proportional to an arbitrary positive, strictly decreasing, and twice differentiable function of the state's energy. The precise shape of the nonlinear map from energies to unnormalized probabilities has to be learned from data together with the parameters of the energy function. As a case study we show that the above generalization of a fully visible Boltzmann machine yields an accurate model of neural activity of retinal ganglion cells. We attribute this success to the model's ability to easily capture distributions whose probabilities span a large dynamic range, a possible consequence of latent variables that globally couple the system. Similar features have recently been observed in many datasets, suggesting that our new method has wide applicability.

Motivation & Objective

• To address the limitation of standard energy-based models in capturing probability distributions with large dynamic ranges, especially when latent variables induce global coupling.
• To develop a flexible, nonparametric method for modeling the nonlinear transformation from energy to unnormalized probability, without assuming a specific functional form.
• To enable joint learning of both the energy function and the nonlinear transformation using unconstrained optimization, improving model expressivity.
• To demonstrate the method's effectiveness on real neural data, particularly retinal ganglion cell activity, where standard models fail to capture the full range of probabilities.

Proposed method

• The model generalizes standard energy-based models by replacing the exponential of the negative energy with an arbitrary strictly decreasing, twice differentiable function V, so that p(s|α,V) ∝ exp(−V(E(s|α))).
• The nonlinearity V is parameterized via a differential equation formulation: V(E) = γ₁ + γ₂∫ₑ₀ᴱ exp(∫ₑ₀ᴱ′ W(E′′) dE′′) dE′, where W is a piecewise-constant function.
• W is expanded in a basis of indicator functions over energy bins, enabling unconstrained optimization of its coefficients β, while γ₁ and γ₂ are learned as scale and shift parameters.
• The model is trained using Persistent Contrastive Divergence with separate learning rates for the coupling matrix J and the nonlinearity parameters (γ, β), ensuring stability.
• The energy range [E₀, E₁] is dynamically updated during training to match the observed energy range under current parameters.
• An approximate likelihood is maximized using a density-of-states estimation algorithm to improve convergence and avoid mode collapse.

Experimental results

Research questions

• RQ1 Can a nonparametric, learnable nonlinearity in the energy-to-probability mapping significantly improve the modeling of complex, high-dynamic-range probability distributions in neural data?
• RQ2 How does the proposed semiparametric model compare to standard pairwise and restricted Boltzmann machine models in capturing the statistics of retinal ganglion cell activity?
• RQ3 To what extent does the learned nonlinearity enable the model to capture the effects of latent variables that globally couple the system?
• RQ4 Can the model achieve better likelihood and statistical fit than parametric models without requiring explicit modeling of hidden units?
• RQ5 Does the nonparametric representation of the nonlinearity allow for stable and efficient joint optimization with the energy function parameters?

Key findings

• The semiparametric model achieved significantly higher log-likelihood on retinal ganglion cell data compared to standard pairwise models and restricted Boltzmann machines, indicating better statistical fit.
• The model successfully captured the full dynamic range of neural activity probabilities, including rare and highly improbable states, which standard models failed to represent.
• The learned nonlinearity V was found to be strongly nonlinear in the low-energy regime, suggesting that the model effectively accounts for global coupling effects from latent variables.
• Joint optimization of the energy function and nonlinearity via Persistent Contrastive Divergence led to stable convergence, with the final model showing improved generalization and reduced mode collapse.
• The use of a piecewise-constant basis for W enabled effective nonparametric modeling of the nonlinearity without imposing restrictive functional forms.
• The final model with Q=12 energy bins and optimized metaparameters E₀ and E₁ achieved the best performance, demonstrating the importance of adaptive energy binning.