[Paper Review] A Survey on Deep Semi-supervised Learning

Xiangli Yang, Zixing Song+2 more|arXiv (Cornell University)|Feb 28, 2021

Multimodal Machine Learning Applications307 references33 citations

TL;DR

This survey categorizes and reviews deep semi-supervised learning (DSSL) methods, covering five main categories and 52 representative methods, with discussion of challenges and future directions.

ABSTRACT

Deep semi-supervised learning is a fast-growing field with a range of practical applications. This paper provides a comprehensive survey on both fundamentals and recent advances in deep semi-supervised learning methods from perspectives of model design and unsupervised loss functions. We first present a taxonomy for deep semi-supervised learning that categorizes existing methods, including deep generative methods, consistency regularization methods, graph-based methods, pseudo-labeling methods, and hybrid methods. Then we provide a comprehensive review of 52 representative methods and offer a detailed comparison of these methods in terms of the type of losses, contributions, and architecture differences. In addition to the progress in the past few years, we further discuss some shortcomings of existing methods and provide some tentative heuristic solutions for solving these open problems.

Motivation & Objective

• Summarize fundamental theory and recent advances in deep semi-supervised learning (DSSL).
• Provide a taxonomy of DSSL methods and map representative methods to categories.
• Compare methods by loss types, architecture, and contributions.
• Identify open problems and suggest heuristic directions for future work.

Proposed method

• Propose a taxonomy of DSSL methods: generative, consistency regularization, graph-based, pseudo-labeling, and hybrid methods.
• Survey 52 representative methods across these categories, comparing losses, architectures, and contributions.
• Discuss open problems and heuristic solutions to advance DSSL.
• Review background, assumptions, and datasets to ground the discussion.
• Highlight connections to related paradigms and applications in vision and NLP.

Experimental results

Research questions

• RQ1 What are the main categories of deep semi-supervised learning and how do they differ in model design and loss formulation?
• RQ2 How do representative DSSL methods compare in terms of losses, architectural choices, and empirical contributions?
• RQ3 What are the key open problems in DSSL and plausible directions to address them?

Key findings

• DSSL methods are classed into five categories: generative, consistency regularization, graph-based, pseudo-labeling, and hybrid methods.
• The survey reviews 52 representative DSSL methods and compares them across loss types and architectures.
• GAN-based and VAE-based generative methods are prominent in leveraging unlabeled data for SSL.
• Consistency regularization and pseudo-labeling strategies are emphasized as effective in many modern approaches.
• The paper discusses shortcomings and proposes heuristic directions for future research.