[Paper Review] On Feature Decorrelation in Self-Supervised Learning

Tianyu Hua, Wenxiao Wang+4 more|arXiv (Cornell University)|May 2, 2021

Domain Adaptation and Few-Shot Learning48 references4 citations

TL;DR

This paper identifies and analyzes two types of representation collapse in self-supervised learning: complete collapse and previously overlooked dimensional collapse, which arises from strong correlations between feature axes. The authors propose feature decorrelation—standardizing the covariance matrix—as a solution, demonstrating empirically that this simple technique significantly improves representation quality across multiple benchmarks, including CIFAR-10, CIFAR-100, and ImageNet.

ABSTRACT

In self-supervised representation learning, a common idea behind most of the state-of-the-art approaches is to enforce the robustness of the representations to predefined augmentations. A potential issue of this idea is the existence of completely collapsed solutions (i.e., constant features), which are typically avoided implicitly by carefully chosen implementation details. In this work, we study a relatively concise framework containing the most common components from recent approaches. We verify the existence of complete collapse and discover another reachable collapse pattern that is usually overlooked, namely dimensional collapse. We connect dimensional collapse with strong correlations between axes and consider such connection as a strong motivation for feature decorrelation (i.e., standardizing the covariance matrix). The gains from feature decorrelation are verified empirically to highlight the importance and the potential of this insight.

Motivation & Objective

• To investigate the existence and causes of representation collapse in self-supervised learning beyond the well-known complete collapse.
• To identify and analyze dimensional collapse—a previously overlooked collapse pattern caused by strong inter-feature correlations.
• To establish a theoretical and empirical link between feature correlation and collapse, motivating feature decorrelation as a solution.
• To evaluate the effectiveness of feature decorrelation in improving downstream performance across diverse self-supervised frameworks and datasets.

Proposed method

• Proposes a concise self-supervised framework (Shuffled-DBN) that isolates core components: data augmentation, encoder, and projector, to study collapse phenomena.
• Introduces a group-wise shuffling strategy to decorrelate features by standardizing the covariance matrix of the projector output.
• Employs a linear evaluation protocol on ImageNet and CIFAR datasets to measure representation utility after decorrelation.
• Uses a projector with learnable parameters and applies batch normalization and weight initialization to stabilize training.
• Varying group size in the shuffling mechanism to control decorrelation strength, enabling ablation studies on decorrelation intensity.
• Employs standard self-supervised contrastive and symmetric loss objectives (e.g., Barlow Twins) within the framework to validate gains from decorrelation.

Experimental results

Research questions

• RQ1 Can dimensional collapse occur in self-supervised representation learning, and what causes it?
• RQ2 Is there a strong connection between high inter-feature correlation and representation collapse?
• RQ3 Does feature decorrelation—specifically, covariance matrix standardization—improve representation quality in self-supervised learning?
• RQ4 How general are the performance gains from decorrelation across different batch sizes, projector dimensions, and datasets?
• RQ5 Can a simple decorrelation technique outperform or match state-of-the-art methods without complex components like momentum encoders or predictors?

Key findings

• Dimensional collapse, caused by strong correlations between feature axes, is a reachable and previously overlooked collapse pattern in self-supervised learning.
• Feature decorrelation via covariance matrix standardization significantly improves linear evaluation accuracy on CIFAR-10, achieving 88.25% top-1 accuracy with a batch size of 32.
• On CIFAR-100, Shuffled-DBN with decorrelation achieves 87.05% kNN accuracy and 88.75% linear accuracy with a group size of 64, outperforming baseline methods.
• On ImageNet with ResNet-50, Shuffled-DBN achieves 65.18% top-1 accuracy after 200 epochs of pretraining, demonstrating strong performance despite a simple framework.
• The performance gains from decorrelation are consistent across different batch sizes and projector dimensions, with gains increasing as decorrelation strength increases.
• The ablation study shows that stronger decorrelation (via larger group size) leads to better performance in both kNN and linear evaluation, confirming the benefit of covariance standardization.