[Paper Review] What Regularized Auto-Encoders Learn from the Data Generating Distribution

Guillaume Alain, Yoshua Bengio|arXiv (Cornell University)|Nov 18, 2012

Generative Adversarial Networks and Image Synthesis24 references101 citations

TL;DR

This paper demonstrates that regularized auto-encoders, particularly denoising and contractive variants, learn the score function (gradient of the log-density) of the data-generating distribution by minimizing a regularized reconstruction error. The key contribution is showing that this training criterion implicitly estimates local density structure—specifically the score and Hessian—without requiring an explicit energy function, enabling approximate MCMC sampling from the estimated distribution.

ABSTRACT

What do auto-encoders learn about the underlying data generating distribution? Recent work suggests that some auto-encoder variants do a good job of capturing the local manifold structure of data. This paper clarifies some of these previous observations by showing that minimizing a particular form of regularized reconstruction error yields a reconstruction function that locally characterizes the shape of the data generating density. We show that the auto-encoder captures the score (derivative of the log-density with respect to the input). It contradicts previous interpretations of reconstruction error as an energy function. Unlike previous results, the theorems provided here are completely generic and do not depend on the parametrization of the auto-encoder: they show what the auto-encoder would tend to if given enough capacity and examples. These results are for a contractive training criterion we show to be similar to the denoising auto-encoder training criterion with small corruption noise, but with contraction applied on the whole reconstruction function rather than just encoder. Similarly to score matching, one can consider the proposed training criterion as a convenient alternative to maximum likelihood because it does not involve a partition function. Finally, we show how an approximate Metropolis-Hastings MCMC can be setup to recover samples from the estimated distribution, and this is confirmed in sampling experiments.

Motivation & Objective

• To clarify what regularized auto-encoders learn about the underlying data-generating distribution.
• To establish a theoretical link between the training criterion of contractive and denoising auto-encoders and the estimation of local density properties such as the score and Hessian.
• To show that minimizing regularized reconstruction error can serve as an alternative to maximum likelihood for unsupervised learning, avoiding the need for partition function computation.
• To demonstrate how an approximate Metropolis-Hastings MCMC can be constructed using the estimated score for sampling from the learned model.

Proposed method

• The paper analyzes a contractive training criterion that applies regularization to the full reconstruction function, not just the encoder, and shows it is equivalent to denoising auto-encoder training with small Gaussian noise.
• It proves that minimizing this criterion leads to a reconstruction function that estimates the score (derivative of the log-density) and Hessian of the data-generating density.
• The method relies on asymptotic analysis under sufficient capacity and data, showing convergence to the true score function regardless of the auto-encoder's parametrization.
• It formulates an approximate Metropolis-Hastings MCMC algorithm where energy differences are estimated using the predicted score, enabling sampling from the learned distribution.
• The approach avoids explicit partition function computation, similar to score matching, and is applicable even when no analytic energy function exists.
• Experiments on artificial datasets validate the method by showing that samples generated via MCMC closely match the training data distribution in 2D projections.

Experimental results

Research questions

• RQ1 What specific aspects of the data-generating distribution do regularized auto-encoders learn?
• RQ2 How does minimizing a regularized reconstruction error relate to estimating the score and Hessian of the log-density?
• RQ3 Can the reconstruction function of an auto-encoder be interpreted as estimating the local mean and density gradient, rather than an energy function?
• RQ4 Is it possible to use the estimated score to perform approximate MCMC sampling from the learned model?
• RQ5 How does this approach compare to maximum likelihood and score matching in terms of training efficiency and distribution estimation?

Key findings

• Minimizing a regularized reconstruction error leads to a reconstruction function that estimates the score (first derivative of the log-density) of the data-generating distribution.
• The method also estimates the Hessian (second derivative of the log-density), capturing local curvature of the density.
• The training criterion is equivalent to denoising auto-encoder training with small Gaussian corruption noise, but with contraction applied to the full reconstruction function.
• The estimated score enables approximate Metropolis-Hastings MCMC sampling, which successfully recovers samples close to the true data distribution in experiments.
• The results hold generally, regardless of the auto-encoder's parametrization, provided sufficient capacity and training data are available.
• The approach avoids the need for partition function computation, making it a viable alternative to maximum likelihood for implicit density estimation.