[Paper Review] An Overview of Deep Semi-Supervised Learning

Yassine Ouali, Céline Hudelot+1 more|arXiv (Cornell University)|Jun 9, 2020

Domain Adaptation and Few-Shot Learning183 references243 citations

TL;DR

This paper provides a comprehensive survey of deep semi-supervised learning (SSL), outlining main SSL categories, core assumptions, evaluation practices, and connections to related topics.

ABSTRACT

Deep neural networks demonstrated their ability to provide remarkable performances on a wide range of supervised learning tasks (e.g., image classification) when trained on extensive collections of labeled data (e.g., ImageNet). However, creating such large datasets requires a considerable amount of resources, time, and effort. Such resources may not be available in many practical cases, limiting the adoption and the application of many deep learning methods. In a search for more data-efficient deep learning methods to overcome the need for large annotated datasets, there is a rising research interest in semi-supervised learning and its applications to deep neural networks to reduce the amount of labeled data required, by either developing novel methods or adopting existing semi-supervised learning frameworks for a deep learning setting. In this paper, we provide a comprehensive overview of deep semi-supervised learning, starting with an introduction to the field, followed by a summarization of the dominant semi-supervised approaches in deep learning.

Motivation & Objective

• Explain the motivations for deep SSL and its data-efficiency goals.
• Summarize the dominant SSL approaches used with deep neural networks.
• Clarify the main assumptions underlying SSL methods and their implications.
• Discuss evaluation practices and realistic baselines for SSL research.
• Outline connections to related areas such as active learning, transfer learning, and domain adaptation.

Proposed method

• Classify SSL methods into major categories: consistency regularization, proxy-label methods, generative models, and graph-based approaches.
• Describe transductive vs inductive learning paradigms and their relevance to SSL.
• Detail representative SSL techniques including Ladder Networks, Pi-Model, Temporal Ensembling, Mean Teacher, Dual Students, SWA/fast-SWA, and Virtual Adversarial Training (VAT).
• Explain core objective formulations that couple unsupervised consistency losses with supervised losses on labeled data.
• Summarize evaluation recommendations for fair comparison and realistic SSL benchmarking.

Experimental results

Research questions

• RQ1 What are the dominant deep SSL approaches and how do they differ conceptually and technically?
• RQ2 What assumptions underlie the effectiveness of SSL methods in deep learning, and when do they hold?
• RQ3 How should SSL methods be evaluated to ensure fair comparisons and real-world relevance?
• RQ4 How are SSL techniques related to and integrated with related areas like active learning, transfer learning, and domain adaptation?

Key findings

• SSL methods leverage unlabeled data to regularize training and improve decision boundaries under cluster, smoothness, and manifold assumptions.
• A range of techniques exist (consistency regularization, proxy labeling, generative, and graph-based) with inductive and transductive variants.
• Representative methods (Ladder Networks, Pi-Model, Temporal Ensembling, Mean Teacher, Dual Students, SWA, VAT) show how unlabeled data can be exploited through consistency and teacher-student frameworks.
• Evaluation best practices include shared implementations, strong supervised baselines, comparisons to transfer learning, and consideration of class distribution mismatch and data splits.
• The survey connects SSL to active learning, transfer learning, domain adaptation, and weakly-supervised learning, highlighting practical considerations such as noisy labels and data quality.