[Paper Review] MoPro: Webly Supervised Learning with Momentum Prototypes

Junnan Li, Caiming Xiong+1 more|arXiv (Cornell University)|Sep 17, 2020

Machine Learning and Data Classification43 references35 citations

TL;DR

MoPro is a webly-supervised representation learning method that uses momentum prototypes to correct noisy labels and detect/remove out-of-distribution samples, achieving state-of-the-art results on WebVision and strong transfer to downstream tasks.

ABSTRACT

We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning, nor the computation unscalability of self-supervised learning. Most existing works on webly-supervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data. We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning. MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset. MoPro also shows superior performance when the pretrained model is transferred to down-stream image classification and detection tasks. It outperforms the ImageNet supervised pretrained model by +10.5 on 1-shot classification on VOC, and outperforms the best self-supervised pretrained model by +17.3 when finetuned on 1\% of ImageNet labeled samples. Furthermore, MoPro is more robust to distribution shifts. Code and pretrained models are available at https://github.com/salesforce/MoPro.

Motivation & Objective

• Address annotation scalability in visual representation learning by leveraging webly-labeled data.
• Develop a noise-robust, efficient learning framework that handles label noise and OOD samples in web data.
• Improve downstream transfer performance for classification and detection using weakly-labeled web images.

Proposed method

• Project images into embeddings with an encoder and normalize to a unit sphere.
• Maintain momentum embeddings and momentum prototypes updated as moving averages.
• Optimize prototypical contrastive loss and instance contrastive loss jointly, plus a cross-entropy loss on pseudo-labels.
• Generate soft pseudo-labels by combining classifier predictions with prototype-based similarity, then convert to hard pseudo-labels with rules for noise correction and OOD removal.
• Update class prototypes as moving averages of embedded samples assigned to each pseudo-label.
• Remove OOD samples from citation-specific losses while keeping them in instance contrastive loss to push them away from in-distribution samples.

Experimental results

Research questions

• RQ1 Can momentum prototypes enable online correction of noisy web labels without additional clean-label data?
• RQ2 Do prototype-based corrections and OOD filtering improve weakly-supervised representation learning and downstream transfer performance?
• RQ3 How does MoPro compare to supervised and self-supervised baselines on upstream webly-labeled data and various downstream tasks?
• RQ4 Is MoPro robust to distribution shifts and capable of better calibration than alternatives?

Key findings

• MoPro achieves state-of-the-art performance on WebVision for upstream webly-supervised learning.
• MoPro substantially improves downstream representation learning for image classification and object detection, outperforming ImageNet-supervised pretraining in certain settings.
• On low-shot transfer tasks, MoPro surpasses self-supervised methods and approaches supervised baselines when data and compute budgets are matched.
• MoPro yields more robust and calibrated models under distribution shifts (ImageNet-R and ImageNet-A) compared to ImageNet-supervised baselines.
• Ablation shows that prototypical loss, instance loss, and prototype-based pseudo-labels each contribute to MoPro’s gains.