[Paper Review] An Empirical Study and Analysis on Open-Set Semi-Supervised Learning

Huixiang Luo, Hao Cheng+5 more|arXiv (Cornell University)|Jan 19, 2021

Domain Adaptation and Few-Shot Learning69 references4 citations

TL;DR

This paper proposes Style Disturbance, a method that enhances both pseudo-labeling (PL) and data-augmentation-based consistency training (DACT) in open-set semi-supervised learning by leveraging out-of-distribution (OOD) samples through style transfer. By reducing the distribution gap between in-distribution and OOD data, the approach achieves state-of-the-art performance on CIFAR-10 and Tiny ImageNet, improving accuracy by 3.35% and 0.92% over prior SOTA methods.

ABSTRACT

Pseudo-labeling (PL) and Data Augmentation-based Consistency Training (DACT) are two approaches widely used in Semi-Supervised Learning (SSL) methods. These methods exhibit great power in many machine learning tasks by utilizing unlabeled data for efficient training. But in a more realistic setting (termed as open-set SSL), where unlabeled dataset contains out-of-distribution (OOD) samples, the traditional SSL methods suffer severe performance degradation. Recent approaches mitigate the negative influence of OOD samples by filtering them out from the unlabeled data. However, it is not clear whether directly removing the OOD samples is the best choice. Furthermore, why PL and DACT could perform differently in open-set SSL remains a mystery. In this paper, we thoroughly analyze various SSL methods (PL and DACT) on open-set SSL and discuss pros and cons of these two approaches separately. Based on our analysis, we propose Style Disturbance to improve traditional SSL methods on open-set SSL and experimentally show our approach can achieve state-of-the-art results on various datasets by utilizing OOD samples properly. We believe our study can bring new insights for SSL research.

Motivation & Objective

• To investigate the robustness of pseudo-labeling (PL) and data-augmentation-based consistency training (DACT) in open-set semi-supervised learning (open-set SSL), where unlabeled data contain out-of-distribution (OOD) samples.
• To analyze why PL and DACT perform differently under OOD conditions and whether filtering OOD samples is optimal.
• To explore the potential of OOD samples as beneficial training signals rather than noise.
• To propose a unified method—Style Disturbance—that enhances both PL and DACT by incorporating OOD-style perturbations.
• To demonstrate that properly utilizing OOD samples can extend performance beyond the traditional upper bound in open-set SSL.

Proposed method

• Style Disturbance uses an AdaIN-based style transfer network to apply OOD-style features to in-distribution (ID) samples, reducing the distribution gap between labeled (ID) and unlabeled (ID+OOD) data.
• For DACT, the method applies style disturbance to both weakly and strongly augmented views of unlabeled data, improving consistency and robustness.
• For PL, OOD-style perturbations are applied directly to the feature representations of ID samples during pseudo-label generation.
• The style transfer network is trained using a content dataset of ID samples and a style dataset of OOD samples, with VGG-19 as the encoder and a learnable decoder.
• A simplified OOD detection module is used for DACT, while for PL, it is assumed that OOD samples are perfectly identified.
• Hyperparameters such as interpolation weights β and ω are randomly sampled from [0, 0.5] during training to increase data diversity.

Experimental results

Research questions

• RQ1 Why does DACT exhibit greater robustness than PL in open-set SSL settings with OOD samples?
• RQ2 Is the removal of OOD samples the optimal strategy for improving model generalization in open-set SSL?
• RQ3 Can OOD samples be leveraged constructively rather than filtered out, and if so, how?
• RQ4 What is the relationship between DACT robustness and the distribution gap between ID and OOD data?
• RQ5 Can a unified method improve both PL and DACT by exploiting OOD samples through style-based data augmentation?

Key findings

• DACT is inherently more robust than PL in open-set SSL, but requires careful and diverse data augmentation strategies to maintain performance.
• The proposed Style Disturbance method improves both PL and DACT-based SSL by reducing the distribution gap between ID and OOD data through OOD-style feature perturbations.
• On CIFAR-10 with 1,000 labeled samples, the method achieves a 3.35% accuracy gain over MTCF (Yu et al., 2020), a prior SOTA method.
• On Tiny ImageNet, the method improves performance by 0.92% over DS3L (Guo et al., 2020), another SOTA baseline.
• Experiments confirm that OOD samples can be beneficial when properly integrated, challenging the assumption that they must be filtered out.
• Ablation studies show that both the style transfer module and the OOD data usage significantly contribute to performance gains, validating the method's design.