[Paper Review] Realistic Evaluation of Deep Semi-Supervised Learning Algorithms

Avital Oliver, Augustus Odena+3 more|arXiv (Cornell University)|Apr 24, 2018

Domain Adaptation and Few-Shot Learning50 references209 citations

TL;DR

The paper provides a unified reimplementation and realistic evaluation of deep SSL methods, showing smaller gains over strong supervised baselines under fair tuning, and highlighting transfer learning and distribution mismatch effects.

ABSTRACT

Semi-supervised learning (SSL) provides a powerful framework for leveraging unlabeled data when labels are limited or expensive to obtain. SSL algorithms based on deep neural networks have recently proven successful on standard benchmark tasks. However, we argue that these benchmarks fail to address many issues that these algorithms would face in real-world applications. After creating a unified reimplementation of various widely-used SSL techniques, we test them in a suite of experiments designed to address these issues. We find that the performance of simple baselines which do not use unlabeled data is often underreported, that SSL methods differ in sensitivity to the amount of labeled and unlabeled data, and that performance can degrade substantially when the unlabeled dataset contains out-of-class examples. To help guide SSL research towards real-world applicability, we make our unified reimplemention and evaluation platform publicly available.

Motivation & Objective

• Assess real-world applicability of deep SSL methods beyond standard benchmarks.
• Provide a unified, reproducible reimplementation to enable fair comparisons across SSL techniques.
• Investigate factors that affect SSL performance: labeled/unlabeled data amounts, class distribution mismatch, and transfer learning.
• Evaluate whether SSL consistently outperforms high-quality supervised baselines under realistic conditions.
• Offer concrete recommendations for evaluating SSL methods in practice.

Proposed method

• Create a unified Wide ResNet (WRN-28-2) and training setup to compare SSL methods fairly.
• Hyperparameter optimization (1000 trials per method) followed by unified, hand-tuned hyperparameters across methods.
• Evaluate on CIFAR-10 (4000 labeled) and SVHN (1000 labeled) with standard unlabeled pools.
• Implement and compare Pi-Model, Mean Teacher, Virtual Adversarial Training (VAT), VAT+EntMin, and Pseudo-Labeling.
• Include baselines: fully supervised with the same model, and transfer learning from ImageNet to CIFAR-10.
• Systematically study effects of unlabeled data quality (distribution mismatch) and data quantities.

Experimental results

Research questions

• RQ1 How do SSL methods perform when evaluated on a common model and training procedure?
• RQ2 Do SSL gains over supervised baselines persist under realistic hyperparameter tuning and baselines?
• RQ3 How does unlabeled data distribution (in- vs out-of-domain) affect SSL performance?
• RQ4 What is the impact of varying labeled and unlabeled data amounts on SSL effectiveness?
• RQ5 Can transfer learning from a large labeled dataset outperform SSL in these settings?

Key findings

• SSL gains over a strong supervised baseline are smaller when using a unified model and extensive hyperparameter tuning.
• On CIFAR-10 with 4000 labels, the best SSL method (VAT + EntMin) achieved 13.13% test error vs 20.26% supervised.
• On SVHN with 1000 labels, VAT achieved 5.63% test error vs 12.83% supervised.
• Transfer learning from ImageNet to CIFAR-10 achieved 12.09% test error, outperforming all SSL methods in this study.
• Unlabeled data from different class distributions can hurt performance compared to not using unlabeled data.
• Performance of SSL methods varies significantly with the amount of labeled and unlabeled data; gains level off beyond ~80k unlabeled samples on SVHN.