[Paper Review] Self-supervised Pretraining of Visual Features in the Wild

Priya Goyal, Mathilde Caron+9 more|arXiv (Cornell University)|Mar 2, 2021

Advanced Image and Video Retrieval TechniquesComputer Science53 references139 citations

TL;DR

SEER pretrained on 1B random images with SwAV and RegNetY (1.3B params) achieves 84.2% top-1 on ImageNet and shows strong few-shot transfer (77.9% with 10% of ImageNet).

ABSTRACT

Recently, self-supervised learning methods like MoCo, SimCLR, BYOL and SwAV have reduced the gap with supervised methods. These results have been achieved in a control environment, that is the highly curated ImageNet dataset. However, the premise of self-supervised learning is that it can learn from any random image and from any unbounded dataset. In this work, we explore if self-supervision lives to its expectation by training large models on random, uncurated images with no supervision. Our final SElf-supERvised (SEER) model, a RegNetY with 1.3B parameters trained on 1B random images with 512 GPUs achieves 84.2% top-1 accuracy, surpassing the best self-supervised pretrained model by 1% and confirming that self-supervised learning works in a real world setting. Interestingly, we also observe that self-supervised models are good few-shot learners achieving 77.9% top-1 with access to only 10% of ImageNet. Code: https://github.com/facebookresearch/vissl

Motivation & Objective

• Demonstrate that self-supervised pretraining on large, uncurated image collections can yield competitive visual representations.
• Assess the scalability of high-capacity architectures (RegNetY) for self-supervised pretraining on billions of images.
• Evaluate transfer performance to ImageNet, low-shot learning, and other downstream tasks.
• Compare self-supervised pretraining on wild data against supervised and weakly supervised baselines.

Proposed method

• Use SwAV online clustering self-supervision to train features without labels.
• Adopt RegNetY architectures (RegNetY-256GF as the focus) for scalable, high-capacity pretraining.
• Train on 1B random, public Instagram images with 8,704-image batch size over 512 V100 GPUs.
• Employ mixed precision, gradient checkpointing, and SyncBatchNorm to enable large-scale training.
• Utilize 6 crops per image and a 3-layer MLP head with 16K prototypes and Sinkhorn iterations for clustering.
• Fine-tune pretrained models on ImageNet and evaluate with standard top-1 accuracy.

Experimental results

Research questions

• RQ1 Can self-supervised learning learn high-quality visual features from billions of random, uncurated images in the wild?
• RQ2 How does model capacity (RegNetY scale) impact downstream performance when pretraining in the wild?
• RQ3 Do self-supervised models pretrained on uncurated data transfer to ImageNet and other benchmarks as effectively as supervised or weakly supervised counterparts?
• RQ4 Is SEER effective in low-shot transfer scenarios, and how does it compare to baselines when limited data is available?
• RQ5 What is the impact of data scale versus number of updates on representation quality?

Key findings

• SEER achieves 84.2% top-1 accuracy on ImageNet after finetuning, surpassing the best previous self-supervised model by 1 percentage point.
• SEER reaches 77.9% top-1 with only 10% of ImageNet in a low-data fine-tuning setting, indicating strong few-shot transfer.
• Self-supervised pretraining on uncurated data transfers competitively to or better than supervised pretraining on ImageNet for several downstream tasks, including COCO detection/segmentation (APBox +1.5–2, APMask +1–2).
• Increasing model capacity yields larger relative gains in transfer performance, especially under low-shot scenarios.
• In linear evaluation on several tasks (iNaturalist, OpenImages, Places, VOC), self-supervised features from SEER outperform supervised ones.