[Paper Review] Dimension Estimation Using Autoencoders

Nitish Bahadur, Randy Paffenroth|arXiv (Cornell University)|Sep 24, 2019

Neural Networks and Applications36 references9 citations

TL;DR

This paper proposes a novel method to estimate intrinsic dimensionality using autoencoders by transforming their latent layer representations into singular value proxies for dimension estimation. It demonstrates that autoencoders yield lower, more variable dimension estimates than PCA on MNIST digits and S&P 500 data, with key design criteria for tuning regularization to stabilize estimates.

ABSTRACT

Dimension Estimation (DE) and Dimension Reduction (DR) are two closely related topics, but with quite different goals. In DE, one attempts to estimate the intrinsic dimensionality or number of latent variables in a set of measurements of a random vector. However, in DR, one attempts to project a random vector, either linearly or non-linearly, to a lower dimensional space that preserves the information contained in the original higher dimensional space. Of course, these two ideas are quite closely linked since, for example, doing DR to a dimension smaller than suggested by DE will likely lead to information loss. Accordingly, in this paper we will focus on a particular class of deep neural networks called autoencoders which are used extensively for DR but are less well studied for DE. We show that several important questions arise when using autoencoders for DE, above and beyond those that arise for more classic DR/DE techniques such as Principal Component Analysis. We address autoencoder architectural choices and regularization techniques that allow one to transform autoencoder latent layer representations into estimates of intrinsic dimension.

Motivation & Objective

• To address the gap in using autoencoders for dimension estimation, a task traditionally dominated by linear methods like PCA.
• To develop design criteria for transforming autoencoder latent representations into valid proxies for singular values used in dimension estimation.
• To compare nonlinear autoencoder-based dimension estimation with linear PCA and Isomap on real-world datasets like MNIST and S&P 500.
• To investigate the sensitivity of autoencoder-based dimension estimates to hyperparameters, particularly regularization strength λ, in financial time series.
• To provide a practical framework for using deep autoencoders in dimension estimation with measurable, interpretable outputs.

Proposed method

• The method uses autoencoders to learn a nonlinear low-dimensional representation of data, treating the latent layer activations as proxies for singular values.
• Regularization via L2 weight decay (λ) is applied to control the magnitude of latent activations, which are then used to estimate intrinsic dimension.
• Dimension is estimated by identifying the number of latent units with activation magnitudes above a threshold (e.g., 1% of cumulative sum) or by capturing 90% of total activation energy.
• The approach extends classical singular value-based dimension estimation (e.g., scree plot analysis) to nonlinear deep autoencoders by replacing SVD outputs with learned latent representations.
• The method is validated on MNIST digits and S&P 500 daily log returns using a sliding window to generate time series of dimension estimates.
• Sensitivity analysis is performed by varying λ and observing changes in latent activation distributions during market stress events.

Experimental results

Research questions

• RQ1 How can autoencoder latent representations be systematically transformed into proxies for singular values to enable dimension estimation?
• RQ2 How do autoencoder-based dimension estimates compare to those from linear methods like PCA and Isomap on MNIST digits?
• RQ3 What is the impact of regularization strength λ on the stability and accuracy of autoencoder-based dimension estimation?
• RQ4 How do dimension estimates from autoencoders respond to market stress in financial time series, such as during sharp market drops?
• RQ5 Can autoencoders detect changes in intrinsic dimensionality of complex, high-dimensional data like S&P 500 returns more sensitively than linear methods?

Key findings

• For MNIST digits, autoencoder-based dimension estimates ranged from 19 to 22 for digits 0–9 (except digit 1), significantly lower than PCA’s 30–32 range.
• On the S&P 500, autoencoder-based dimension estimates showed higher variance than PCA, suggesting greater sensitivity to market dynamics and tuning parameters.
• During market stress events—such as the 6% drop on August 8, 2011—autoencoder latent activations showed notable shifts, indicating responsiveness to volatility.
• The dimension estimate for digit 1 was consistently lower (13–19) across methods, suggesting its simpler structure may reflect lower intrinsic dimensionality.
• Increasing regularization strength λ systematically reduced the magnitude of latent activations, affecting the number of significant components identified in the dimension estimation process.
• The time series of dimension estimates from autoencoders captured market stress periods more sharply than PCA, indicating potential for detecting systemic risk through dimensionality changes.