[Paper Review] Barlow Twins: Self-Supervised Learning via Redundancy Reduction

Jure Žbontar, Jing Li+3 more|arXiv (Cornell University)|Mar 4, 2021

Domain Adaptation and Few-Shot Learning48 references777 citations

TL;DR

Barlow Twins introduces a self-supervised objective that decorrelates components of twin embeddings while enforcing invariance, enabling effective representations without large batches or asymmetric network designs. It matches or exceeds state-of-the-art on several ImageNet benchmarks, especially with high-dimensional embeddings.

ABSTRACT

Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks. A successful approach to SSL is to learn embeddings which are invariant to distortions of the input sample. However, a recurring issue with this approach is the existence of trivial constant solutions. Most current methods avoid such solutions by careful implementation details. We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks fed with distorted versions of a sample, and making it as close to the identity matrix as possible. This causes the embedding vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors. The method is called Barlow Twins, owing to neuroscientist H. Barlow's redundancy-reduction principle applied to a pair of identical networks. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors. Barlow Twins outperforms previous methods on ImageNet for semi-supervised classification in the low-data regime, and is on par with current state of the art for ImageNet classification with a linear classifier head, and for transfer tasks of classification and object detection.

Motivation & Objective

• Motivate self-supervised learning (SSL) for visual representations without annotations.
• Propose a principled objective that avoids collapsed solutions without requiring asymmetry.
• Leverage redundancy reduction to decorrelate embedding components while preserving invariance.
• Demonstrate robustness to batch size and explore high-dimensional embedding benefits.

Proposed method

• Compute two distorted views of each image and pass them through identical networks.
• Form a cross-correlation matrix between twin outputs and target it to be close to the identity.
• Decompose loss into an invariance term (diagonal) and a redundancy reduction term (off-diagonal) with a trade-off parameter lambda.
• Use a ResNet-50 encoder with a three-layer 8192-d projector; normalize embeddings along the batch; optimize with LARS over large-scale ImageNet pretraining.
• Show that the method works with batch sizes as small as 256 and benefits from high-dimensional embeddings.

Experimental results

Research questions

• RQ1 Can a simple, symmetric twin-network objective avoid collapse without asymmetry or large batch requirements?
• RQ2 How does redundancy reduction affect embedding decorrelation and downstream transfer performance?
• RQ3 What is the impact of embedding dimensionality and projector depth on SSL quality?
• RQ4 Is the method robust to batch size and augmentation choices compared to InfoNCE-based approaches?

Key findings

• Barlow Twins achieves competitive ImageNet linear-top-1 accuracy (73.2%) with a ResNet-50 encoder.
• Under 1% and 10% labeled settings, the method matches or slightly surpasses competing SSL methods in semi-supervised ImageNet.
• Transfer results show competitive performance on Places-205, VOC07, and iNaturalist18 with linear fixed representations.
• For object detection and instance segmentation, Barlow Twins performs comparably or better than several SOTA methods.
• Ablations show both invariance and redundancy terms are necessary; the method remains robust to small batch sizes and benefits from high embedding dimensionality.
• Increasing projector dimensionality continues to improve performance, unlike some other SSL methods.