[Paper Review] An Embarrassingly Simple Baseline for Imbalanced Semi-Supervised Learning

Hao Chen, Yue Fan+6 more|arXiv (Cornell University)|Nov 20, 2022

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes SimiS, a simple yet highly effective baseline for imbalanced semi-supervised learning that reduces class imbalance by augmenting labeled data with pseudo-labels based on the frequency difference from the most frequent class. It achieves state-of-the-art performance, outperforming prior methods by 12.8%, 13.6%, and 16.7% on CIFAR100-LT, FOOD101-LT, and ImageNet127 respectively, with faster convergence and improved pseudo-label accuracy.

ABSTRACT

Semi-supervised learning (SSL) has shown great promise in leveraging unlabeled data to improve model performance. While standard SSL assumes uniform data distribution, we consider a more realistic and challenging setting called imbalanced SSL, where imbalanced class distributions occur in both labeled and unlabeled data. Although there are existing endeavors to tackle this challenge, their performance degenerates when facing severe imbalance since they can not reduce the class imbalance sufficiently and effectively. In this paper, we study a simple yet overlooked baseline -- SimiS -- which tackles data imbalance by simply supplementing labeled data with pseudo-labels, according to the difference in class distribution from the most frequent class. Such a simple baseline turns out to be highly effective in reducing class imbalance. It outperforms existing methods by a significant margin, e.g., 12.8%, 13.6%, and 16.7% over previous SOTA on CIFAR100-LT, FOOD101-LT, and ImageNet127 respectively. The reduced imbalance results in faster convergence and better pseudo-label accuracy of SimiS. The simplicity of our method also makes it possible to be combined with other re-balancing techniques to improve the performance further. Moreover, our method shows great robustness to a wide range of data distributions, which holds enormous potential in practice. Code will be publicly available.

Motivation & Objective

• To address the challenge of severe class imbalance in semi-supervised learning, where both labeled and unlabeled data exhibit skewed class distributions.
• To propose a simple yet highly effective baseline method that mitigates imbalance without complex re-weighting or architecture changes.
• To demonstrate that a straightforward approach—supplementing labeled data with pseudo-labels based on class frequency differences—can significantly outperform existing state-of-the-art methods.
• To show that the method is robust across diverse data distributions and compatible with other re-balancing techniques.

Proposed method

• SimiS reduces class imbalance by generating and incorporating pseudo-labels into the labeled set based on the relative frequency difference from the most frequent class.
• The method uses a simple heuristic: for each class, the number of pseudo-labels added is proportional to the inverse of its frequency relative to the most frequent class.
• It applies standard SSL training with the augmented labeled set, maintaining the same training pipeline as baseline SSL methods.
• The approach does not require additional hyperparameters or complex loss re-weighting, relying only on class frequency statistics.
• It leverages the inherent consistency of pseudo-labeling in SSL to improve label quality and model generalization.
• The method is compatible with existing SSL frameworks and can be combined with other re-balancing techniques for further gains.

Experimental results

Research questions

• RQ1 Can a simple, non-learned heuristic for pseudo-labeling significantly reduce class imbalance in semi-supervised learning?
• RQ2 How does SimiS compare to state-of-the-art methods in terms of performance and convergence speed under severe class imbalance?
• RQ3 To what extent is SimiS robust across different data distributions and imbalance levels?
• RQ4 Can SimiS be effectively combined with other re-balancing techniques to further improve performance?

Key findings

• SimiS outperforms previous state-of-the-art methods by 12.8% on CIFAR100-LT, 13.6% on FOOD101-LT, and 16.7% on ImageNet127 under long-tailed data distributions.
• The method achieves faster convergence due to reduced class imbalance in the training data.
• Pseudo-label accuracy improves significantly because the balanced data distribution leads to more reliable predictions during training.
• SimiS demonstrates strong robustness across a wide range of data imbalance levels and distributions.
• The method's simplicity allows seamless integration with other re-balancing techniques, enabling further performance gains.
• The approach achieves superior performance without requiring complex loss functions, re-weighting, or architectural modifications.