[Paper Review] Billion-scale semi-supervised learning for image classification

İsmet Zeki Yalnız, Hervé Jeǵou+3 more|arXiv (Cornell University)|May 2, 2019

Advanced Image and Video Retrieval TechniquesComputer Science42 references331 citations

TL;DR

This paper scales semi-supervised learning to billions of unlabeled images using a teacher/student pipeline to improve image classification accuracy on large architectures like ResNet-50 and ResNeXt, achieving state-of-the-art results on ImageNet and other tasks.

ABSTRACT

This paper presents a study of semi-supervised learning with large convolutional networks. We propose a pipeline, based on a teacher/student paradigm, that leverages a large collection of unlabelled images (up to 1 billion). Our main goal is to improve the performance for a given target architecture, like ResNet-50 or ResNext. We provide an extensive analysis of the success factors of our approach, which leads us to formulate some recommendations to produce high-accuracy models for image classification with semi-supervised learning. As a result, our approach brings important gains to standard architectures for image, video and fine-grained classification. For instance, by leveraging one billion unlabelled images, our learned vanilla ResNet-50 achieves 81.2% top-1 accuracy on the ImageNet benchmark.

Motivation & Objective

• Motivate and assess semi-supervised learning at web-wide scale for image classification.
• Develop a scalable teacher/student pipeline to exploit billions of unlabeled images.
• Identify factors that influence performance and provide practical recommendations for high-accuracy models.

Proposed method

• Train a high-capacity teacher model on labeled data to generate predictions on a massive unlabeled set.
• For each class, rank unlabeled images by teacher predictions and select top-K examples to form a new labeled set hat{D}.
• Train a student model on hat{D}, potentially with a different architecture to reduce test-time complexity.
• Fine-tune the student on the original labeled data to correct labeling noise and improve generalization.
• Optionally pre-train the teacher with weak supervision on large weakly-labeled data before ranking unlabeled images.

Experimental results

Research questions

• RQ1 How does a teacher/student semi-supervised pipeline perform when scaling to billions of unlabeled images across large CNN architectures?
• RQ2 What are the critical factors (teacher strength, unlabeled data scale, sampling parameters K and P) that maximize accuracy gains?
• RQ3 Can semi-supervised learning with unlabeled data provide gains over fully supervised training for standard architectures on ImageNet and other benchmarks?
• RQ4 What is the role of fine-tuning on true labels after leveraging inferred labels from unlabeled data?
• RQ5 Does weakly supervised pre-training of the teacher further improve results?

Key findings

• Using a teacher trained on labeled data to select top-K unlabeled examples per class yields strong gains for a fixed architecture.
• Fine-tuning the student on true labeled data after learning from hat{D} is crucial for achieving high accuracy.
• Larger unlabeled datasets and longer pre-training improve performance, with diminishing returns beyond saturation for some models.
• Increasing teacher capacity up to a point boosts student accuracy; beyond that, gains plateau due to ImageNet’s size limits.
• Self-training (teacher and student same) also yields improvements, but teacher/student configurations typically perform better for a given target architecture.
• Pre-training the teacher with weak supervision on IG-1B-Targeted before ranking unlabeled data further boosts ImageNet results, achieving state-of-the-art for several architectures.