[Paper Review] Self-ensembling for visual domain adaptation

Geoffrey French, Michał Mackiewicz+1 more|UEA Digital Repository (University of East Anglia)|Jun 16, 2017

Domain Adaptation and Few-Shot Learning25 references63 citations

TL;DR

This paper adapts the mean teacher self-ensembling framework for visual domain adaptation, introducing domain-specific training with source/target batches, confidence thresholding, and class balancing to achieve state-of-the-art results on several benchmarks including VisDA-2017.

ABSTRACT

This paper explores the use of self-ensembling for visual domain adaptation problems. Our technique is derived from the mean teacher variant (Tarvainen et al., 2017) of temporal ensembling (Laine et al;, 2017), a technique that achieved state of the art results in the area of semi-supervised learning. We introduce a number of modifications to their approach for challenging domain adaptation scenarios and evaluate its effectiveness. Our approach achieves state of the art results in a variety of benchmarks, including our winning entry in the VISDA-2017 visual domain adaptation challenge. In small image benchmarks, our algorithm not only outperforms prior art, but can also achieve accuracy that is close to that of a classifier trained in a supervised fashion.

Motivation & Objective

• Motivate reducing labeling needs via semi-supervised and unsupervised domain adaptation for visual tasks.
• Extend mean teacher self-ensembling to handle separate labeled source and unlabeled target domains.
• Improve training stability and performance with confidence thresholding and class balancing in challenging domain shifts.
• Demonstrate effectiveness across small image benchmarks and the VisDA-2017 domain adaptation challenge.

Proposed method

• Base the approach on mean teacher semi-supervised learning where a student network learns from a teacher that is an exponential moving average of the student.
• Use separate source and target pathways to process domain-specific data with batch normalization statistics adapted per domain.
• Apply the standard supervised loss on labeled source data and an unsupervised self-ensembling loss between student and teacher predictions on target data.
• Replace Gaussian ramp-up with confidence thresholding to decide when to apply the self-ensembling loss for unlabeled target samples.
• Introduce a class balance loss to mitigate degenerate minima caused by imbalanced target class distributions.
• Incorporate data augmentation schemes (Gaussian noise, translations, flips, affine transforms) and evaluate their impact on different benchmarks.

Experimental results

Research questions

• RQ1 Can self-ensembling be effectively transferred from semi-supervised learning to unsupervised domain adaptation with labeled source and unlabeled target data?
• RQ2 Do domain-specific batch normalization, confidence-based weighting, and class balancing improve adaptation performance on challenging datasets?
• RQ3 What augmentation and training strategies yield state-of-the-art results across small image benchmarks and larger domain adaptation tasks such as VisDA-2017?
• RQ4 How close can domain-adapted self-ensembling approach the performance of supervised learning on the target domain?

Key findings

• Achieved state-of-the-art results on several small image domain adaptation benchmarks with MT+TF, MT+CT, MT+CT+TF, and MT+CT+TFA configurations.
• MT+TF alone delivered strong results on several tasks; adding confidence thresholding (CT) stabilized training and improved performance in some benchmarks (e.g., STLCIFAR, Syn-digits to SVHN).
• Confidence thresholding at a 0.968 threshold improved stability and performance, acting as a filter to emphasize high-confidence teacher predictions.
• Class balance loss mitigated degenerate minima caused by target class imbalance (notably MNIST→SVHN), balancing target predictions toward uniform class distribution.
• On VisDA-2017, using a pretrained ResNet-152 with test-time augmentation and ensemble predictions yielded competitive competition results, including high validation/test scores across configurations.
• Data augmentation choices significantly affected performance; translations/ flips aided some benchmarks but affine augmentations could hurt others depending on data domain.