[Paper Review] Semi-supervised Long-tailed Recognition using Alternate Sampling

Bo Liu, Haoxiang Li+3 more|arXiv (Cornell University)|May 1, 2021

Domain Adaptation and Few-Shot Learning27 references4 citations

TL;DR

This paper proposes a novel semi-supervised long-tailed recognition framework that decouples feature embedding and classifier training, using class-balanced sampling for the classifier on labeled data and random sampling for embedding on both labeled and unlabeled data. The method achieves state-of-the-art accuracy, especially on few-shot classes, by iteratively refining pseudo-labels and mitigating bias from low-quality pseudo labels in tail classes.

ABSTRACT

Main challenges in long-tailed recognition come from the imbalanced data distribution and sample scarcity in its tail classes. While techniques have been proposed to achieve a more balanced training loss and to improve tail classes data variations with synthesized samples, we resort to leverage readily available unlabeled data to boost recognition accuracy. The idea leads to a new recognition setting, namely semi-supervised long-tailed recognition. We argue this setting better resembles the real-world data collection and annotation process and hence can help close the gap to real-world scenarios. To address the semi-supervised long-tailed recognition problem, we present an alternate sampling framework combining the intuitions from successful methods in these two research areas. The classifier and feature embedding are learned separately and updated iteratively. The class-balanced sampling strategy has been implemented to train the classifier in a way not affected by the pseudo labels' quality on the unlabeled data. A consistency loss has been introduced to limit the impact from unlabeled data while leveraging them to update the feature embedding. We demonstrate significant accuracy improvements over other competitive methods on two datasets.

Motivation & Objective

• Address the gap between standard semi-supervised learning benchmarks and real-world data collection, where both labeled and unlabeled data follow long-tailed distributions.
• Overcome the challenge of poor pseudo-label quality for tail classes in semi-supervised long-tailed recognition, where standard methods fail due to class imbalance.
• Develop a training framework that decouples classifier and feature embedding learning to stabilize training and improve robustness to noisy pseudo-labels.
• Improve recognition accuracy on long-tailed datasets by leveraging unlabeled data without exacerbating bias toward head classes.

Proposed method

• Decouple the recognition model into a feature embedding network and a classifier, trained separately with different sampling strategies.
• Use class-balanced sampling for the classifier on the labeled subset to ensure fair supervision and reduce bias toward head classes.
• Apply random sampling for feature embedding training on both labeled and unlabeled data to avoid bias from potentially incorrect pseudo-labels.
• Introduce a consistency loss between forward passes of the same unlabeled sample under different augmentations to improve feature generalization.
• Implement an alternate learning scheme: first train the classifier on labeled data, then update the feature embedding using pseudo-labeled unlabeled data, and iterate.
• Use pseudo-labels from the current model to supervise the embedding training, but avoid updating the classifier on pseudo-labeled data to prevent error propagation.

Experimental results

Research questions

• RQ1 Can semi-supervised learning techniques be effectively adapted to long-tailed recognition when both labeled and unlabeled data follow the same long-tailed distribution?
• RQ2 How does the quality of pseudo-labels for tail classes affect model performance in semi-supervised long-tailed recognition?
• RQ3 Can decoupling classifier and feature embedding training with different sampling strategies reduce bias and improve robustness to noisy pseudo-labels?
• RQ4 Does alternate learning between classifier and embedding training improve performance on few-shot classes compared to joint training or end-to-end fine-tuning?

Key findings

• The proposed method achieves 70.1% overall accuracy on CIFAR-10-SSLT (imbalance factor 100), significantly outperforming the best baseline by over 6 percentage points.
• On few-shot classes, the method achieves 56.5% accuracy, a substantial improvement over the baseline (14.1% with R+R and 37.6% with C+R), demonstrating robustness to pseudo-label noise.
• Abllation studies show that training the classifier on pseudo-labeled data (even with class-balanced sampling) harms performance, especially on few-shot classes, due to error propagation.
• Pseudo-label accuracy on the unlabeled subset increases from 41.8% to 57.8% on few-shot splits over 5 training loops, confirming the effectiveness of the alternate learning scheme.
• Removing unsupervised training of the feature embedding leads to a significant performance drop (worse than the initialized model), proving that unsupervised learning is essential for improvement.
• The method maintains high performance on many-shot classes (89.6% overall) while drastically improving few-shot accuracy, indicating balanced learning across class cardinalities.