[Paper Review] Imbalanced Semi-supervised Learning with Bias Adaptive Classifier

Renzhen Wang, Xixi Jia+3 more|arXiv (Cornell University)|Jul 28, 2022

Imbalanced Data Classification Techniques4 citations

TL;DR

This paper proposes a bias adaptive classifier for imbalanced semi-supervised learning that decouples unbiased label prediction from bias assimilation using a bi-level optimization framework. By integrating a lightweight residual-based bias attractor with a linear classifier, the method mitigates performance degradation in minority classes—achieving up to 60% higher recall on the most imbalanced classes compared to FixMatch.

ABSTRACT

Pseudo-labeling has proven to be a promising semi-supervised learning (SSL) paradigm. Existing pseudo-labeling methods commonly assume that the class distributions of training data are balanced. However, such an assumption is far from realistic scenarios and thus severely limits the performance of current pseudo-labeling methods under the context of class-imbalance. To alleviate this problem, we design a bias adaptive classifier that targets the imbalanced SSL setups. The core idea is to automatically assimilate the training bias caused by class imbalance via the bias adaptive classifier, which is composed of a novel bias attractor and the original linear classifier. The bias attractor is designed as a light-weight residual network and optimized through a bi-level learning framework. Such a learning strategy enables the bias adaptive classifier to fit imbalanced training data, while the linear classifier can provide unbiased label prediction for each class. We conduct extensive experiments under various imbalanced semi-supervised setups, and the results demonstrate that our method can be applied to different pseudo-labeling models and is superior to current state-of-the-art methods.

Motivation & Objective

• Address the performance degradation of pseudo-labeling methods in semi-supervised learning when class distributions are imbalanced.
• Overcome the issue of biased pseudo-labels that amplify class imbalance during training.
• Develop a model-agnostic framework that protects the linear classifier from training bias while maintaining representation learning.
• Enable effective use of unlabeled data in imbalanced setups by learning an unbiased classifier through a novel bi-level optimization strategy.

Proposed method

• Introduce a bias adaptive classifier composed of a linear classifier for unbiased predictions and a bias attractor for assimilating training bias.
• Design the bias attractor as a lightweight residual network to model complex data bias without overfitting.
• Employ a bi-level learning framework: lower-level optimizes the full network with labeled and unlabeled data; upper-level tunes the bias attractor using a re-sampled, class-balanced subset of labeled data.
• Decouple the linear classifier's optimization from the biased data distribution, allowing it to generalize equally across all classes.
• Use online class-balanced sampling in the upper-level objective to ensure the linear classifier remains unbiased during training.
• Integrate the proposed classifier into existing pseudo-labeling frameworks such as FixMatch without architectural changes, ensuring model-agnostic applicability.

Experimental results

Research questions

• RQ1 How does class imbalance in labeled data affect the performance of existing pseudo-labeling methods in semi-supervised learning?
• RQ2 Can a classifier be trained to remain unbiased despite learning from imbalanced data with noisy pseudo-labels?
• RQ3 Does a bi-level optimization framework effectively decouple bias assimilation from unbiased prediction in imbalanced SSL?
• RQ4 To what extent can a lightweight bias attractor improve minority class generalization in semi-supervised learning?
• RQ5 How does the proposed method compare to state-of-the-art approaches under varying levels of class imbalance in both labeled and unlabeled data?

Key findings

• The proposed L2AC framework improves minority class recall by approximately 60% compared to FixMatch on CIFAR-10 with an imbalance ratio of 100.
• The linear classifier trained with L2AC closely approximates the predicted class distribution of an upper-bound model trained on full labeled data, indicating strong unbiased generalization.
• t-SNE visualization confirms that L2AC produces more discriminative representations, especially for tail classes, which are often misclassified by baseline models.
• Abllation studies show that the bi-level training strategy is critical—removing it leads to a significant performance drop, even when the bias attractor is retained.
• The method achieves consistent improvements across multiple datasets and imbalance ratios, including reversed imbalance settings where labeled data are rare and unlabeled data are balanced.
• The bias attractor alone provides marginal gains, but when combined with bi-level optimization, it enables substantial performance gains, demonstrating the importance of the hierarchical training strategy.