[Paper Review] Improved Baselines with Momentum Contrastive Learning

Xinlei Chen, Haoqi Fan+2 more|arXiv (Cornell University)|Mar 9, 2020

Multimodal Machine Learning Applications17 references2,109 citations

TL;DR

This paper embeds SimCLR-inspired improvements (MLP projection head and stronger data augmentation) into MoCo, achieving stronger unsupervised baselines that surpass SimCLR and do not require large training batches. It also analyzes memory and time costs on 8 GPUs.

ABSTRACT

Contrastive unsupervised learning has recently shown encouraging progress, e.g., in Momentum Contrast (MoCo) and SimCLR. In this note, we verify the effectiveness of two of SimCLR's design improvements by implementing them in the MoCo framework. With simple modifications to MoCo---namely, using an MLP projection head and more data augmentation---we establish stronger baselines that outperform SimCLR and do not require large training batches. We hope this will make state-of-the-art unsupervised learning research more accessible. Code will be made public.

Motivation & Objective

• Motivate stronger, accessible unsupervised baselines within the MoCo framework.
• Investigate whether SimCLR-inspired improvements (MLP projection head, stronger augmentation) transfer to MoCo.
• Quantify performance gains in ImageNet linear classification and VOC object detection transfer.
• Assess computational cost and practicality of MoCo-based improvements on mainstream hardware.

Proposed method

• Implement an MLP projection head in MoCo v2 (2-layer MLP with 2048-d hidden layer).
• Incorporate stronger data augmentation including blur as in SimCLR.
• Explore cosine learning rate scheduling for MoCo and compare against baseline.
• Evaluate on ImageNet linear classification (1-crop, 224) and VOC07+12 object detection transfer with a Faster R-CNN detector.
• Compare against MoCo v1 and SimCLR under varying epochs and batch sizes.
• Provide ablations showing the impact of MLP, augmentation, and scheduling on performance.

Experimental results

Research questions

• RQ1 Does adding an MLP projection head to MoCo improve downstream transfer performance?
• RQ2 Does stronger data augmentation (including blur) improve MoCo-based representations?
• RQ3 Can MoCo achieve competitive or superior results to SimCLR without extremely large batch sizes?
• RQ4 What are the trade-offs in training time and memory when adopting MoCo v2 with these improvements?

Key findings

• MoCo v2 with an MLP projection head and stronger augmentation yields higher ImageNet linear accuracy than MoCo v1 and SimCLR under the same settings.
• Using the optimal temperature (tau) with MLP improves from 60.6% to 66.2% in ImageNet linear accuracy.
• The extra augmentation alone raises ImageNet accuracy to 63.4%, while the MLP alone achieves 66.2% (with optimal tau); combining both yields 67.3%.
• With 800 pre-training epochs, MoCo v2 reaches 71.1% ImageNet accuracy, outperforming SimCLR’s 69.3% with 1000 epochs.
• On VOC object detection, MoCo v2 variants show robust transfer, with AP50/AP/AP75 comparable to or better than baseline MoCo v1 and SimCLR at similar training regimes.
• MoCo’s negative-key queue decouples batch size from negatives, enabling strong performance on standard 8-GPU hardware without requiring large-scale TPU batches.