[Paper Review] Rethinking Re-Sampling in Imbalanced Semi-Supervised Learning

Ju He, Adam Kortylewski+5 more|arXiv (Cornell University)|Jun 1, 2021

Domain Adaptation and Few-Shot Learning33 references15 citations

TL;DR

This paper proposes Bi-Sampling (BiS), a novel two-stage re-sampling strategy for class-imbalanced semi-supervised learning that decouples feature extractor and classifier training. BiS progressively shifts from unbalanced sampling (to preserve representation learning) to balanced sampling (to improve classifier generalization), achieving state-of-the-art performance on CIFAR-10 and CIFAR-100 with gains of up to 7.5% over baselines.

ABSTRACT

Semi-Supervised Learning (SSL) has shown its strong ability in utilizing unlabeled data when labeled data is scarce. However, most SSL algorithms work under the assumption that the class distributions are balanced in both training and test sets. In this work, we consider the problem of SSL on class-imbalanced data, which better reflects real-world situations. In particular, we decouple the training of the representation and the classifier, and systematically investigate the effects of different data re-sampling techniques when training the whole network including a classifier as well as fine-tuning the feature extractor only. We find that data re-sampling is of critical importance to learn a good classifier as it increases the accuracy of the pseudo-labels, in particular for the minority classes in the unlabeled data. Interestingly, we find that accurate pseudo-labels do not help when training the feature extractor, rather contrariwise, data re-sampling harms the training of the feature extractor. This finding is against the general intuition that wrong pseudo-labels always harm the model performance in SSL. Based on these findings, we suggest to re-think the current paradigm of having a single data re-sampling strategy and develop a simple yet highly effective Bi-Sampling (BiS) strategy for SSL on class-imbalanced data. BiS implements two different re-sampling strategies for training the feature extractor and the classifier and integrates this decoupled training into an end-to-end framework. In particular, BiS progressively changes the data distribution during training such that in the beginning the feature extractor is trained effectively, while towards the end of the training the data is re-balanced such that the classifier is trained reliably. We benchmark our proposed bi-sampling strategy extensively on popular datasets and achieve state-of-the-art performances.

Motivation & Objective

• To investigate the impact of data re-sampling on feature extractor and classifier learning in class-imbalanced semi-supervised learning.
• To challenge the conventional paradigm of using a single re-sampling strategy throughout training.
• To develop a method that optimizes representation learning and classifier calibration separately using distinct sampling strategies.
• To achieve state-of-the-art performance on imbalanced semi-supervised benchmarks like CIFAR-10 and CIFAR-100.

Proposed method

• BiS employs two distinct re-sampling strategies: a random sampler for the feature extractor and a mean sampler for the classifier.
• The method progressively transitions from the random sampler to the mean sampler during training using a decay schedule for the sampling weight α.
• The sampling weight α decays according to a parabolic function over training steps, favoring unbalanced data early and balanced data later.
• The classifier is fine-tuned with re-sampling after joint training of the feature extractor and classifier, decoupling the learning process.
• The keep probability q controls the strength of re-sampling for unlabeled data, with ablation showing optimal performance at intermediate values.
• The framework is integrated end-to-end and requires no additional training techniques like mixup.

Experimental results

Research questions

• RQ1 How does data re-sampling affect the learning of feature extractors versus classifiers in imbalanced semi-supervised learning?
• RQ2 Does re-sampling harm or help representation learning when pseudo-labels are biased toward head classes?
• RQ3 Can a single re-sampling strategy effectively balance both representation learning and classifier calibration in long-tailed SSL?
• RQ4 What is the optimal schedule for transitioning between different re-sampling strategies during training?
• RQ5 Can a decoupled training strategy with distinct sampling for feature extractor and classifier outperform joint re-sampling?

Key findings

• Re-sampling harms feature extractor training despite biased pseudo-labels, indicating that representation learning benefits from unbalanced data.
• Classifier performance significantly improves with re-sampling, especially for minority classes, even when pseudo-labels are inaccurate.
• BiS achieves a 7.5% absolute improvement over FixMatch baseline on CIFAR-10 with β=2 and λ=100.
• On CIFAR-100, BiS achieves a 4% absolute gain over FixMatch and 1% over CReST+ under similar settings.
• The parabolic decay schedule for α outperforms linear, cosine, and equal decay by approximately 8% in performance.
• The ablation study shows that extreme values of keep probability q (0 or 1) degrade performance, with optimal values in the middle range.