[Paper Review] From Principal Subspaces to Principal Components with Linear Autoencoders

Elad Plaut|arXiv (Cornell University)|Apr 26, 2018

Generative Adversarial Networks and Image Synthesis15 references83 citations

TL;DR

The paper shows how to recover PCA loading vectors from the weights of a linear autoencoder, enabling exact PCA via linear autoencoders and discussing properties, efficiency, and online applicability. It proves that the first m loading vectors are the first m left singular vectors of the autoencoder’s weight matrix and demonstrates empirical alignment with PCA on MNIST and CUB-200-2011 datasets.

ABSTRACT

The autoencoder is an effective unsupervised learning model which is widely used in deep learning. It is well known that an autoencoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function trains weights that span the same subspace as the one spanned by the principal component loading vectors, but that they are not identical to the loading vectors. In this paper, we show how to recover the loading vectors from the autoencoder weights.

Motivation & Objective

• Motivate PCA as a linear transform that maximizes variance and minimizes reconstruction error.
• Show that a linear autoencoder's weights span the principal subspace but do not equal loading vectors, and propose recovering loading vectors from weights.
• Propose and validate a method to recover the PCA loading vectors from the autoencoder weights, yielding unique, decorrelated, ordered coordinates.
• Demonstrate the practicality of this approach on large, high-dimensional datasets and online learning scenarios.

Proposed method

• Formulate PCA via centered data and define loading vectors as eigenvectors of the centered covariance matrix.
• Show that a linear autoencoder with a single hidden layer and squared error loss leads to a weight optimization problem equivalent to projecting onto the principal subspace.
• Prove that the loading vectors can be recovered by taking the left singular vectors of the autoencoder’s weight matrix W2 (or W1 via W2’s pseudoinverse).
• Argue that the recovered loading vectors yield a diagonal covariance in the transformed coordinates, matching PCA properties (sorted variances, decorrelation).
• Discuss online training suitability and advantages over standard PCA, including not requiring data centering and compatibility with large-scale data.

Experimental results

Research questions

• RQ1 Can the loading vectors of PCA be recovered from the weights of a linear autoencoder?
• RQ2 Is the recovered loading set unique and does it preserve PCA properties (decorrelation, ordering, nesting) across dimensionality reductions?
• RQ3 Does training a linear autoencoder with standard optimizers and regularization yield results equivalent to PCA when loading vectors are recovered from weights?
• RQ4 How does this approach perform on high-dimensional, large-scale datasets and in online learning settings?

Key findings

• The first m loading vectors of Y coincide with the first m left singular vectors of W2 (and equivalently W1 via pseudoinverses).
• Recovering loading vectors from autoencoder weights yields a unique, decorrelated, variances-ordered coordinate system and nested solutions for different target dimensions.
• Empirical results on MNIST show that left singular vectors of W2 closely approximate PCA loading vectors up to sign; covariance in transformed coordinates becomes diagonal with descending diagonal elements.
• On large, high-dimensional datasets like CUB-200-2011, PCA loading vectors can be recovered from W2, achieving approximately diagonal covariance in transformed space, validating scalability.