[Paper Review] Investigating Power laws in Deep Representation Learning

Arna Ghosh, Arnab Kumar Mondal+2 more|arXiv (Cornell University)|Feb 11, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes using the power-law decay coefficient α of the eigenspectrum of deep neural network representations as a label-free proxy for representation quality. By analyzing feature covariance matrices across diverse architectures and self-supervised learning objectives, the authors find that α ≈ 1 strongly correlates with superior downstream generalization and robustness to label noise, offering a scalable, label-free metric for evaluating self-supervised representations without requiring downstream tasks.

ABSTRACT

Representation learning that leverages large-scale labelled datasets, is central to recent progress in machine learning. Access to task relevant labels at scale is often scarce or expensive, motivating the need to learn from unlabelled datasets with self-supervised learning (SSL). Such large unlabelled datasets (with data augmentations) often provide a good coverage of the underlying input distribution. However evaluating the representations learned by SSL algorithms still requires task-specific labelled samples in the training pipeline. Additionally, the generalization of task-specific encoding is often sensitive to potential distribution shift. Inspired by recent advances in theoretical machine learning and vision neuroscience, we observe that the eigenspectrum of the empirical feature covariance matrix often follows a power law. For visual representations, we estimate the coefficient of the power law, $α$, across three key attributes which influence representation learning: learning objective (supervised, SimCLR, Barlow Twins and BYOL), network architecture (VGG, ResNet and Vision Transformer), and tasks (object and scene recognition). We observe that under mild conditions, proximity of $α$ to 1, is strongly correlated to the downstream generalization performance. Furthermore, $α\approx 1$ is a strong indicator of robustness to label noise during fine-tuning. Notably, $α$ is computable from the representations without knowledge of any labels, thereby offering a framework to evaluate the quality of representations in unlabelled datasets.

Motivation & Objective

• To investigate whether the eigenspectrum of deep neural network representations follows a power law, particularly in self-supervised learning.
• To determine if the power-law coefficient α correlates with downstream generalization performance and robustness to distribution shift or label noise.
• To develop a label-free evaluation metric for representation quality that does not require access to downstream labels.
• To explore the relationship between α and model robustness during fine-tuning under noisy supervision.
• To assess whether α can serve as a principled, scalable proxy for representation quality across architectures and learning objectives.

Proposed method

• Compute the empirical feature covariance matrix Σ_N(f) from deep network activations on large-scale unlabelled datasets.
• Compute the eigenspectrum of Σ_N(f) and fit a power law λ_i ∝ i^(-α) to estimate the decay coefficient α.
• Evaluate α across different network architectures (VGG, ResNet, Vision Transformer), learning objectives (supervised, SimCLR, Barlow Twins, BYOL), and downstream tasks (object and scene recognition).
• Use α as a proxy for representation quality and correlate it with downstream accuracy on STL-10 and ImageNet.
• Perform fine-tuning with 15% label noise to assess robustness, tracking changes in α and accuracy over training epochs.
• Compare α values across models and training objectives to identify patterns in representation quality independent of labels.

Experimental results

Research questions

• RQ1 Does the eigenspectrum of deep neural network representations follow a power law, and what is the value of the decay coefficient α?
• RQ2 How does α correlate with downstream generalization performance across different architectures and learning objectives?
• RQ3 Is α a reliable predictor of robustness to label noise during fine-tuning?
• RQ4 Can α be used as a label-free metric to evaluate representation quality in self-supervised learning?
• RQ5 Does α ≈ 1 indicate a fundamental property of representations that enables better generalization and robustness?

Key findings

• The power-law decay coefficient α of the eigenspectrum is strongly correlated with downstream classification accuracy on STL-10 and ImageNet, with performance peaking when α ≈ 1.
• Models pre-trained with self-supervised objectives (e.g., Barlow Twins, SimCLR) exhibit α ≈ 1 and achieve higher accuracy (e.g., 91.81% on STL-10) compared to supervised pre-training.
• Under 15% label noise during fine-tuning, models with α closer to 1 (e.g., Barlow Twins) show significantly better robustness, with only ~3.36% accuracy drop, compared to ~8.20% for supervised pre-training.
• As fine-tuning progresses and validation accuracy improves, α approaches 1 from both sides, indicating a dynamic relationship between performance and spectral structure.
• The ViT architecture exhibits α > 1 in early layers due to global receptive fields, suggesting higher redundancy and lower effective rank in early representations.
• α is computable from representations without labels, making it a practical, scalable metric for evaluating self-supervised representations in unlabelled datasets.