[논문 리뷰] An Overview of Deep Semi-Supervised Learning

Yassine Ouali, Céline Hudelot외 1명|arXiv (Cornell University)|2020. 06. 09.

Domain Adaptation and Few-Shot Learning참고 문헌 183인용 수 243

한 줄 요약

이 논문은 심층 반지도 학습(SSL)에 대한 포괄적인 조사로, 주요 SSL 범주, 핵심 가정, 평가 관행 및 관련 주제와의 연결을 개략적으로 제시한다.

ABSTRACT

Deep neural networks demonstrated their ability to provide remarkable performances on a wide range of supervised learning tasks (e.g., image classification) when trained on extensive collections of labeled data (e.g., ImageNet). However, creating such large datasets requires a considerable amount of resources, time, and effort. Such resources may not be available in many practical cases, limiting the adoption and the application of many deep learning methods. In a search for more data-efficient deep learning methods to overcome the need for large annotated datasets, there is a rising research interest in semi-supervised learning and its applications to deep neural networks to reduce the amount of labeled data required, by either developing novel methods or adopting existing semi-supervised learning frameworks for a deep learning setting. In this paper, we provide a comprehensive overview of deep semi-supervised learning, starting with an introduction to the field, followed by a summarization of the dominant semi-supervised approaches in deep learning.

연구 동기 및 목표

• Explain the motivations for deep SSL and its data-efficiency goals.
• Summarize the dominant SSL approaches used with deep neural networks.
• Clarify the main assumptions underlying SSL methods and their implications.
• Discuss evaluation practices and realistic baselines for SSL research.
• Outline connections to related areas such as active learning, transfer learning, and domain adaptation.

제안 방법

• Classify SSL methods into major categories: consistency regularization, proxy-label methods, generative models, and graph-based approaches.
• Describe transductive vs inductive learning paradigms and their relevance to SSL.
• Detail representative SSL techniques including Ladder Networks, Pi-Model, Temporal Ensembling, Mean Teacher, Dual Students, SWA/fast-SWA, and Virtual Adversarial Training (VAT).
• Explain core objective formulations that couple unsupervised consistency losses with supervised losses on labeled data.
• Summarize evaluation recommendations for fair comparison and realistic SSL benchmarking.

실험 결과

연구 질문

• RQ1 What are the dominant deep SSL approaches and how do they differ conceptually and technically?
• RQ2 What assumptions underlie the effectiveness of SSL methods in deep learning, and when do they hold?
• RQ3 How should SSL methods be evaluated to ensure fair comparisons and real-world relevance?
• RQ4 How are SSL techniques related to and integrated with related areas like active learning, transfer learning, and domain adaptation?

주요 결과

• SSL 방법은 비표지 데이터를 활용해 학습을 규제하고 군집, 매끄러움, 매니폴드 가정 하에서 의사결정 경계를 향상시킨다.
• 일관성 규제, 프록시 레이블링, 생성형, 그래프 기반 등 다양한 기술이 존재하며, 귀납적 및 트랜스덕티브 변형이 있다.
• 대표적 방법들(Ladder Networks, Pi-Model, Temporal Ensembling, Mean Teacher, Dual Students, SWA, VAT)은 비표지 데이터를 일관성과 교수-학생 구조를 통해 활용하는 방법을 보여준다.
• 공정한 비교와 현실적인 SSL 벤치마킹을 위한 평가 모범 사례로, 공유 구현, 강력한 감독 기초선, 전이 학습과의 비교, 클래스 분포 불일치 및 데이터 분할 고려가 포함된다.
• 이 조사서는 SSL을 활성 학습, 전이 학습, 도메인 적응 및 약지도 학습과 연결하며, 노이즈가 있는 레이블 및 데이터 품질과 같은 실용적 고려사항을 강조한다.