[Paper Review] Auto-encoders: reconstruction versus compression

Yann Ollivier|arXiv (Cornell University)|Mar 30, 2014

Generative Adversarial Networks and Image SynthesisComputer Science17 references17 citations

TL;DR

This paper establishes a theoretical link between auto-encoder training for reconstruction error and data compression via minimum description length (MDL). It shows that minimizing codelength—equivalent to optimal compression—requires adding information-theoretic regularization terms to reconstruction error, which correspond to a denoising criterion from features to output and determine an optimal, data-dependent noise level for denoising auto-encoders.

ABSTRACT

We discuss the similarities and differences between training an auto-encoder to minimize the reconstruction error, and training the same auto-encoder to compress the data via a generative model. Minimizing a codelength for the data using an auto-encoder is equivalent to minimizing the reconstruction error plus some correcting terms which have an interpretation as either a denoising or contractive property of the decoding function. These terms are related but not identical to those used in denoising or contractive auto-encoders [Vincent et al. 2010, Rifai et al. 2011]. In particular, the codelength viewpoint fully determines an optimal noise level for the denoising criterion.

Motivation & Objective

• To investigate the relationship between minimizing reconstruction error in auto-encoders and minimizing data codelength for optimal compression.
• To determine whether standard auto-encoder training implicitly minimizes codelength, and if not, what modifications are needed.
• To derive a variational upper bound on codelength that incorporates both reconstruction error and information-theoretic regularization.
• To connect the codelength framework with denoising and contractive auto-encoders, showing how MDL determines optimal hyperparameters like noise level.
• To explore how including output variances as learnable parameters improves compression bounds and aligns with relative error minimization.

Proposed method

• Uses the minimum description length (MDL) principle to frame auto-encoding as data compression, where the goal is to minimize the codelength of the data.
• Applies a variational bound to derive an upper bound on codelength that depends on reconstruction error and additional regularization terms.
• Derives a tight codelength bound valid for continuous feature spaces (Proposition 2), which approaches the true codelength when the encoder and decoder are probabilistically inverse.
• Introduces a denoising criterion from features to output, with an optimal noise level derived from the inverse Hessian of the reconstruction function.
• Reinterprets contractive auto-encoders through the MDL lens, showing that codelength minimization penalizes large output derivatives from features, with a penalty term based on row norms of the Jacobian.
• Proposes including output variances as parameters to improve compression bounds, leading to a reconstruction objective involving log of squared errors plus quantization noise.

Experimental results

Research questions

• RQ1 Does minimizing reconstruction error in auto-encoders correspond to minimizing the codelength of the data under the MDL principle?
• RQ2 What additional regularization terms are required to make reconstruction error training equivalent to codelength minimization?
• RQ3 Can the MDL framework determine an optimal, data-dependent noise level for denoising auto-encoders, and how does it differ from fixed noise levels?
• RQ4 How does the codelength minimization criterion relate to contractive auto-encoders, and what is the resulting penalty structure on the Jacobian?
• RQ5 What is the impact of modeling output variances as learnable parameters on the codelength bound and reconstruction error?

Key findings

• Minimizing reconstruction error alone does not minimize codelength; additional information-theoretic terms are required to achieve optimal compression.
• The codelength bound includes a denoising penalty from features to output, which is distinct from the input-to-feature denoising used in prior work.
• The optimal noise level for denoising is determined by the inverse Hessian of the reconstruction function, and should be set differently for each data sample.
• The codelength framework leads to a penalty on large derivatives from features to output, which differs from the Frobenius norm used in standard contractive auto-encoders.
• Including output variances as parameters results in a reconstruction objective that minimizes the logarithm of the squared errors, aligning with relative error minimization and improving compression bounds.
• The variational codelength bound is tight when the encoder and decoder are probabilistically inverse, and provides a principled way to train auto-encoders for compression.