[Paper Review] Better Mixing via Deep Representations

Yoshua Bengio, Grégoire Mesnil+2 more|arXiv (Cornell University)|Jul 18, 2012

Generative Adversarial Networks and Image Synthesis32 references207 citations

TL;DR

This paper proposes that deeper neural representations improve Markov chain Monte Carlo (MCMC) mixing by better disentangling underlying factors of variation, leading to more uniform coverage of high-density manifolds. Experiments show that sampling at higher layers yields faster mixing, better sample quality, and improved generative performance, reconciling efficient mixing with strong discriminative ability through disentangled representations.

ABSTRACT

It has previously been hypothesized, and supported with some experimental evidence, that deeper representations, when well trained, tend to do a better job at disentangling the underlying factors of variation. We study the following related conjecture: better representations, in the sense of better disentangling, can be exploited to produce faster-mixing Markov chains. Consequently, mixing would be more efficient at higher levels of representation. To better understand why and how this is happening, we propose a secondary conjecture: the higher-level samples fill more uniformly the space they occupy and the high-density manifolds tend to unfold when represented at higher levels. The paper discusses these hypotheses and tests them experimentally through visualization and measurements of mixing and interpolating between samples.

Motivation & Objective

• To investigate whether deeper representations lead to faster-mixing Markov chains in generative modeling.
• To test the hypothesis that better disentanglement of underlying factors of variation in deep representations improves mixing efficiency.
• To examine whether higher-level representations expand the volume of high-quality samples while maintaining or improving discriminative performance.
• To reconcile improved mixing with strong classification performance, countering the intuition that closer class manifolds reduce discriminability.

Proposed method

• The authors train deep autoencoders (CAE) and deep belief networks (DBN) on MNIST and TFD datasets to learn hierarchical representations.
• They evaluate mixing by measuring the number of distinct classes visited over multiple MCMC sampling steps at different representation levels.
• They assess sample quality via log-likelihood estimation using Parzen windows on generated samples from deeper layers.
• They analyze manifold structure by interpolating between examples and adding isotropic noise to representations, measuring likelihoods and coverage.
• They use linear SVMs and fine-tuned MLPs to evaluate discriminative performance on concatenated or higher-layer representations.
• They visualize local convex hulls and balls to study how representation depth affects the geometry of data manifolds.

Experimental results

Research questions

• RQ1 Does sampling from deeper representations lead to faster mixing in MCMC chains compared to lower-level representations?
• RQ2 To what extent do deeper representations disentangle underlying factors of variation, particularly class factors?
• RQ3 Does the increased volume of high-quality samples at deeper layers contradict better discriminative performance, and if so, how is this reconciled?
• RQ4 How does the geometry of data manifolds—specifically unfolding and expansion—change across representation levels?
• RQ5 Can better mixing at higher layers be leveraged to improve training efficiency in deep generative models that rely on MCMC for gradient estimation?

Key findings

• Sampling from deeper layers of CAE and DBN models resulted in significantly faster mixing, with 20 samples visiting more classes than 100 samples from lower layers.
• The log-likelihood of generated samples improved substantially at deeper layers: DBN-2 achieved a log-likelihood of 1908.80 ± 65.94, compared to 604 ± 14.67 for DBN-1.
• Interpolating between examples at higher layers produced higher-likelihood samples, indicating better manifold coverage and smoother transitions.
• Adding isotropic noise to deeper representations yielded more plausible samples, confirming that high-density regions are better expanded and uniformly covered at higher levels.
• Despite improved mixing and volume expansion, deeper representations maintained or improved discriminative performance, with a 0.81% error rate on MNIST using fine-tuned MLPs on CAE-2 features.
• The results support the hypothesis that better disentanglement of class factors at deeper layers reconciles improved generative sampling with strong classification ability.