[Paper Review] CReST: A Class-Rebalancing Self-Training Framework for Imbalanced Semi-Supervised Learning

Chen Wei, Kihyuk Sohn+3 more|arXiv (Cornell University)|Feb 18, 2021

Imbalanced Data Classification Techniques45 references33 citations

TL;DR

CReST introduces a class-rebalancing self-training framework (and a progressive variant CReST+) to improve imbalanced semi-supervised learning by preferentially adding minority-class pseudo-labels and progressively aligning distributions, yielding strong gains over state-of-the-art SSL baselines on long-tailed datasets.

ABSTRACT

Semi-supervised learning on class-imbalanced data, although a realistic problem, has been under studied. While existing semi-supervised learning (SSL) methods are known to perform poorly on minority classes, we find that they still generate high precision pseudo-labels on minority classes. By exploiting this property, in this work, we propose Class-Rebalancing Self-Training (CReST), a simple yet effective framework to improve existing SSL methods on class-imbalanced data. CReST iteratively retrains a baseline SSL model with a labeled set expanded by adding pseudo-labeled samples from an unlabeled set, where pseudo-labeled samples from minority classes are selected more frequently according to an estimated class distribution. We also propose a progressive distribution alignment to adaptively adjust the rebalancing strength dubbed CReST+. We show that CReST and CReST+ improve state-of-the-art SSL algorithms on various class-imbalanced datasets and consistently outperform other popular rebalancing methods. Code has been made available at https://github.com/google-research/crest.

Motivation & Objective

• Motivate and address the under-studied problem of SSL under class imbalance where labeled and unlabeled data share imbalanced distributions.
• Investigate the bias of existing SSL methods on imbalanced data and identify high precision but low recall on minority classes.
• Propose a self-training framework (CReST) that rebalances the labeled set by sampling pseudo-labeled minority-class samples more aggressively.
• Extend with distribution alignment (CReST+) to progressively align predictions to counter model bias during self-training.
• Demonstrate improved recall for minority classes and overall accuracy on CIFAR-LT and ImageNet127 benchmarks across various label fractions and imbalance ratios.

Proposed method

• Observe that minority-class pseudo-label precision is high despite low recall for minority classes in SSL.
• Iteratively retrain a baseline SSL model and expand the labeled set with selectively sampled pseudo-labeled data from unlabeled data, biased toward minority classes based on estimated class distribution.
• Use a sampling rate mu_l proportional to ((N_{L+1-l}/N_1))^alpha to determine how many pseudo-labeled samples from class l to add (alpha > 0 favors minority classes).
• Introduce progressive distribution alignment (DA) with temperature scaling t to control reproportioning of pseudo-labels across generations (CReST+).
• Progressively decrease t across generations to balance precision and recall while increasing the influence of rebalancing in later generations.
• Benchmark with FixMatch and MixMatch on CIFAR-LT and ImageNet127 showing improvements over baselines and prior methods.

Experimental results

Research questions

• RQ1 How does SSL performance degrade on class-imbalanced data and can minority-class precision be leveraged to improve recall?
• RQ2 Can a class-rebalancing self-training loop improve the recall and overall accuracy of SSL methods on imbalanced labeled/unlabeled data?
• RQ3 Does progressive distribution alignment (CReST+) further enhance pseudo-label quality and reduce model bias across generations?
• RQ4 How do the proposed methods perform on long-tailed CIFAR (CIFAR10-LT, CIFAR100-LT) and large-scale ImageNet127 datasets under varying label fractions?
• RQ5 Do CReST and CReST+ outperform state-of-the-art imbalanced SSL methods such as DARP and standard rebalancing approaches?

Key findings

• CReST consistently improves over baseline SSL methods across evaluated imbalances and label fractions, with gains up to 11.8% in accuracy on CIFAR-LT benchmarks.
• CReST+ further boosts performance via progressive distribution alignment, achieving additional gains over CReST in multiple settings.
• On CIFAR-LT (gamma up to 200) and 10–30% labeled data, CReST/CReST+ outperform FixMatch and MixMatch baselines and several rebalancing baselines.
• On ImageNet127 with 10% labeled data, CReST+ yields a 7.9% recall improvement over the baseline.
• Ablation studies show that nonzero alpha (class-rebalancing sampling) is crucial for gains, and progressively scheduled temperature t in CReST+ balances precision and recall effectively.
• CReST+ combined with logits-adjustment post-processing (LA) provides further improvements, yielding best results in some comparisons.