[Paper Review] Revisiting Contrastive Methods for Unsupervised Learning of Visual Representations

Wouter Van Gansbeke, Simon Vandenhende+2 more|arXiv (Cornell University)|Jun 10, 2021

Domain Adaptation and Few-Shot LearningComputer Science61 references34 citations

TL;DR

This paper analyzes dataset biases in contrastive self-supervised learning (MoCo-based) and shows that standard augmentations transfer across object-centric, scene-centric, uniform or long-tailed and domain-specific data; it also introduces additional invariances (multi-crop, stronger augmentations, nearest neighbors) to boost representations and demonstrates spatially structured features from multi-crop.

ABSTRACT

Contrastive self-supervised learning has outperformed supervised pretraining on many downstream tasks like segmentation and object detection. However, current methods are still primarily applied to curated datasets like ImageNet. In this paper, we first study how biases in the dataset affect existing methods. Our results show that current contrastive approaches work surprisingly well across: (i) object- versus scene-centric, (ii) uniform versus long-tailed and (iii) general versus domain-specific datasets. Second, given the generality of the approach, we try to realize further gains with minor modifications. We show that learning additional invariances -- through the use of multi-scale cropping, stronger augmentations and nearest neighbors -- improves the representations. Finally, we observe that MoCo learns spatially structured representations when trained with a multi-crop strategy. The representations can be used for semantic segment retrieval and video instance segmentation without finetuning. Moreover, the results are on par with specialized models. We hope this work will serve as a useful study for other researchers. The code and models are available at https://github.com/wvangansbeke/Revisiting-Contrastive-SSL.

Motivation & Objective

• Assess how dataset biases (object-centric vs scene-centric, uniform vs long-tailed, general vs domain-specific) affect contrastive self-supervised learning.
• Explore simple, model-agnostic modifications to improve representations without new pretext tasks.
• Investigate whether architectural/data cues yield spatially structured representations useful for dense tasks.
• Evaluate transferability of learned representations across a broad set of downstream tasks.

Proposed method

• Build on MoCo with a ResNet-50 backbone and projection head; use a memory bank with a momentum encoder to provide negatives.
• Evaluate pretraining on diverse datasets (ImageNet, COCO, OpenImages, BDD100K) and assess transfer to linear, segmentation, detection, video segmentation, and depth tasks.
• Study dataset bias effects by comparing object-centric vs scene-centric data and uniform vs long-tailed distributions.
• Propose three invariance-enhancing additions: multi-crop transform, stronger augmentations, and online nearest-neighbor positives.
• Introduce constrained multi-crop and multi-crop with reduced momentum to improve training efficiency.
• Propose kNN-MoCo: a nearest-neighbors auxiliary loss using a second queue from backbone features to regularize representations.

Experimental results

Research questions

• RQ1 Do MoCo-like contrastive SSL methods degrade when pretrained on scene-centric or long-tailed datasets compared to object-centric datasets?
• RQ2 Can simple, dataset-agnostic augmentations transfer across diverse downstream tasks without custom domain priors?
• RQ3 Do additional invariances (multi-crop, stronger augmentations, nearest neighbors) improve transfer performance and enable dense representations without labeled data?
• RQ4 Can multi-crop learning yield spatially structured representations useful for dense prediction tasks without finetuning?

Key findings

• MoCo-based pretraining performs robustly across object-centric and scene-centric datasets, and across uniform and long-tailed distributions.
• Increasing data diversity (COCO/OpenImages) during pretraining can match or exceed ImageNet-pretrained results on several tasks when finetuned.
• Multi-crop and constrained multi-crop substantially improve transfer performance and enable learning of spatially structured representations that support dense tasks without fine-tuning.
• Stronger augmentations alone may not always help; combining standard and stronger augmentations yields better VOC segmentation performance.
• Nearest-neighbors augmentation (kNN-MoCo) further improves linear-classification transfer and maintains gains across multiple tasks.
• Final models pretrained with these invariances achieve competitive or superior results to specialized methods on several tasks, including video instance segmentation and semantic segment retrieval.