[Paper Review] Why (and When and How) Contrastive Divergence Works

Ian E. Fellows|arXiv (Cornell University)|May 3, 2014

Complex Network Analysis Techniques10 references3 citations

TL;DR

This paper provides a rigorous theoretical foundation for contrastive divergence (CD) in exponential-family models with intractable normalizing constants, framing CD as a variational approximation and justifying its moment conditions. It shows that CD works best when MCMC kernels efficiently revisit highly dependent subsets of the state space—such as dyads connected to the same node—enabling faster convergence and improved inference over pseudo-likelihood, especially in high-dimensional models like exponential-family random graph models (ERGMs).

ABSTRACT

Contrastive divergence (CD) is a promising method of inference in high dimensional distributions with intractable normalizing constants, however, the theoretical foundations justifying its use are somewhat shaky. This document proposes a framework for understanding CD inference, how/when it works, and provides multiple justifications for the CD moment conditions, including framing them as a variational approximation. Algorithms for performing inference are discussed and are applied to social network data using an exponential-family random graph models (ERGM). The framework also provides guidance about how to construct MCMC kernels providing good CD inference, which turn out to be quite different from those used typically to provide fast global mixing.

Motivation & Objective

• To address theoretical weaknesses in contrastive divergence (CD) by providing a principled foundation for its use in models with intractable normalizing constants.
• To justify the CD moment conditions through multiple lenses, including variational approximation, rather than relying on heuristic assumptions.
• To guide the design of effective MCMC kernels that enable CD to outperform pseudo-likelihood in high-dimensional exponential-family models.
• To demonstrate through simulation that CD with well-constructed kernels achieves estimates close to maximum likelihood, especially for transitivity and network structure terms.

Proposed method

• Proposes a novel augmented divergence framework that reinterprets CD as a variational approximation to the true KL divergence minimization.
• Reframes the CD objective function to include a variational lower bound, providing theoretical justification for the CD moment conditions.
• Introduces the Node-s MCMC kernel, which selects a node and then updates s dyads connected to it, improving mixing over random or sequential scans.
• Employs Newton-like updates for faster convergence in parameter estimation, replacing slower gradient-based methods.
• Uses a geometrically weighted edgewise shared partner (GWESP) statistic and an offset for maximum degree constraints in ERGMs to model realistic network dependencies.
• Applies the framework to social network data using exponential-family random graph models (ERGMs), comparing CD with pseudo-likelihood and MLE.

Experimental results

Research questions

• RQ1 Why does contrastive divergence work in practice despite weak theoretical justification?
• RQ2 When does CD provide meaningful improvements over pseudo-likelihood in high-dimensional exponential-family models?
• RQ3 How should MCMC kernels be designed to maximize CD performance, especially in high-dimensional state spaces?
• RQ4 What is the theoretical basis for the CD moment condition g(y) = E_Tq(g(Y))?
• RQ5 How can CD be made computationally efficient while maintaining accuracy close to maximum likelihood?

Key findings

• CD inference significantly outperforms pseudo-likelihood for transitivity in ERGMs, with GWESP estimates under CD being within one standard deviation of MLE, unlike MPLE which was over 3 standard deviations off.
• Node-s MCMC kernels that focus on dyads connected to the same node achieve near-MLE performance with k ≥ 2048 steps, even though the subset dimensionality is only 1269.
• Random scan Gibbs samplers fail to improve over pseudo-likelihood even with 16,384 steps, confirming that global mixing is insufficient when chains do not revisit dependent regions efficiently.
• The Newton-like update scheme enables faster convergence than gradient updates, validating its use in CD inference for complex models.
• CD with well-designed kernels reduces bias in mean value parameters for edges, nodeMatch, and isolates, bringing estimates closer to MLE than pseudo-likelihood.
• Theoretical analysis confirms that CD works best when MCMC kernels revisit highly dependent parts of the state space quickly, rather than relying on long chains to reach equilibrium.