[Paper Review] Two SVDs produce more focal deep learning representations

Hinrich Schuetze, Christian Scheible|arXiv (Cornell University)|Jan 16, 2013

Domain Adaptation and Few-Shot Learning16 references5 citations

TL;DR

This paper proposes a two-stage SVD-based method (SVD2) that enhances deep learning representations by applying two consecutive singular value decompositions: the first for denoising, the second to rotate the representation into a more 'focal' configuration. The key contribution is that SVD2 produces representations with significantly reduced inter-dimension correlations—approximately 3× lower on average—indicating improved focality, which may support better generalization and interpretability in neural networks.

ABSTRACT

A key characteristic of work on deep learning and neural networks in general is that it relies on representations of the input that support generalization, robust inference, domain adaptation and other desirable functionalities. Much recent progress in the field has focused on efficient and effective methods for computing representations. In this paper, we propose an alternative method that is more efficient than prior work and produces representations that have a property we call focality -- a property we hypothesize to be important for neural network representations. The method consists of a simple application of two consecutive SVDs and is inspired by Anandkumar (2012).

Motivation & Objective

• To develop a more efficient and effective method for generating deep learning representations with improved structural properties.
• To investigate whether applying two consecutive SVDs can produce representations with higher focality—a property hypothesized to support better generalization and interpretability.
• To evaluate representation quality using diagnostic measures independent of downstream tasks, focusing on correlation structure and dimensionality efficiency.
• To explore the potential of stacking SVDs as a nonlinear, efficient alternative to standard deep learning representation learning methods.
• To compare SVD2 with traditional deep learning and other representation learning techniques in terms of representation quality and computational efficiency.

Proposed method

• Apply the first SVD (SVD1) to a base $n \times d$ representation matrix $C$ to obtain a rank-$k$ approximation $C_k = USV^T$, using $US$ as a $k$-dimensional representation.
• Normalize each $k$-dimensional vector from SVD1 to unit length to emphasize relative differences over absolute magnitudes.
• Apply a second SVD (SVD2) to the normalized $n \times k$ matrix $C'$, resulting in $C' = U'S'V'^T$, and use $U'S'$ as the final $k$-dimensional representation.
• Introduce 'focality' as a measure of representation quality: a representation is more focal if fewer dimensions are needed to achieve a given classification accuracy threshold.
• Use correlation histograms of dimension pairs to diagnose focality—lower average correlation indicates higher focality.
• Evaluate discrimination performance by determining the minimal number of dimensions required to achieve a threshold accuracy in distinguishing word pairs.

Experimental results

Research questions

• RQ1 Does applying two consecutive SVDs produce representations with higher focality than a single SVD?
• RQ2 How does the correlation structure between representation dimensions differ between SVD1 and SVD2?
• RQ3 Can SVD2 representations achieve better discrimination performance with fewer active dimensions compared to SVD1?
• RQ4 How does stacking a second SVD (2LAYER) compare to a single-layer SVD2 in terms of representation quality?
• RQ5 To what extent does the SVD2 method outperform or complement existing deep learning and nonlinear dimensionality reduction techniques in representation quality and efficiency?

Key findings

• SVD2 representations exhibit a 3× reduction in average inter-dimension correlation compared to SVD1, as shown by a 0.5 log10-scale shift in correlation coefficient histograms.
• The distribution of correlation coefficients for SVD2 is shifted leftward by approximately 0.5 on a log10 scale, indicating that correlations are reduced by a factor of about 3.
• SVD2 representations require fewer active dimensions to achieve a given classification accuracy threshold, confirming higher focality compared to SVD1.
• 2LAYER SVD2 representations (two stacked SVDs) outperform 1LAYER SVD2 representations in terms of focality and discrimination performance.
• The method is computationally efficient and nonlinear due to normalization between SVD applications, offering a viable alternative to standard deep learning initialization methods.
• The proposed mSVD method, though simple, shows potential as a lightweight, interpretable alternative to complex deep learning or autoencoding-based representation learning.