[Paper Review] Exploring Simple Siamese Representation Learning

Xinlei Chen, Kaiming He|arXiv (Cornell University)|Nov 20, 2020

Domain Adaptation and Few-Shot LearningComputer Science36 references414 citations

TL;DR

SimSiam shows that simple Siamese networks can learn meaningful representations without negative pairs, large batches, or momentum encoders, thanks to stop-gradient; achieves competitive ImageNet and transfer results.

ABSTRACT

Siamese networks have become a common structure in various recent models for unsupervised visual representation learning. These models maximize the similarity between two augmentations of one image, subject to certain conditions for avoiding collapsing solutions. In this paper, we report surprising empirical results that simple Siamese networks can learn meaningful representations even using none of the following: (i) negative sample pairs, (ii) large batches, (iii) momentum encoders. Our experiments show that collapsing solutions do exist for the loss and structure, but a stop-gradient operation plays an essential role in preventing collapsing. We provide a hypothesis on the implication of stop-gradient, and further show proof-of-concept experiments verifying it. Our "SimSiam" method achieves competitive results on ImageNet and downstream tasks. We hope this simple baseline will motivate people to rethink the roles of Siamese architectures for unsupervised representation learning. Code will be made available.

Motivation & Objective

• Motivate a rethinking of Siamese architectures in unsupervised representation learning.
• Investigate whether simple Siamese setups can avoid collapsing without common safeguards (negatives, large batches, momentum).
• Characterize the role of stop-gradient and other architectural components in preventing collapse.
• Provide empirical evidence and a hypothesis linking SimSiam to an EM-like optimization process.

Proposed method

• Two augmented views of each image are processed by a shared encoder f (backbone + projection MLP).
• A predictor M is applied to one branch to produce p, while the other branch yields z; the loss minimizes negative cosine similarity between p and stopgrad(z) and between the symmetric pair.
• Stop-gradient is applied to one branch to prevent gradients flowing through its view, preventing collapse.
• Loss is symmetrized as the average of the two asymmetric terms.
• Training uses standard SGD with a 512-batch, synchronized BN, and a 3-layer projection MLP plus a 2-layer predictor.
• Evaluations rely on ImageNet linear classification on frozen representations, with additional ablations on batch size, BN, similarity functions, and symmetrization.

Experimental results

Research questions

• RQ1 Can a Siamese network learn meaningful representations without negative samples, large batches, or momentum encoders?
• RQ2 What mechanism (e.g., stop-gradient) prevents collapse in a simple Siamese setup?
• RQ3 How do architectural choices like the predictor, batch normalization, and similarity measure affect performance and collapse tendency?
• RQ4 How do the learned representations transfer to downstream tasks beyond ImageNet?
• RQ5 What theoretical viewpoint can explain SimSiam’s behavior and its relation to existing methods?

Key findings

• Stop-gradient is essential to prevent collapsing, enabling nontrivial ImageNet linear accuracy (67%+ in experiments) even without negatives or momentum encoders.
• Predictor MLP is necessary for success; removing it or fixing its parameters markedly reduces accuracy.
• A wide range of batch sizes work well, indicating no need for very large batches; 64–2048 batch sizes produced competitive results.
• Batch normalization in the projection and prediction heads improves optimization and accuracy; BN in the output of the projection head is beneficial, while BN on the predictor output is not.
• Cosine similarity and cross-entropy-like similarity can both work, with cosine generally performing better in their experiments.
• Symmetrization improves accuracy but is not required for preventing collapse; stop-gradient is the key collapse-preventing mechanism.