[Paper Review] Robust Long-Tailed Learning under Label Noise

Tong Wei, Jiang-Xin Shi+2 more|arXiv (Cornell University)|Aug 26, 2021

Machine Learning and Data Classification41 references24 citations

TL;DR

The paper introduces RoLT, a robust framework for learning under long-tailed distributions with label noise, using a prototypical noise detector and soft pseudo-labeling to improve generalisation, especially for tail classes.

ABSTRACT

Long-tailed learning has attracted much attention recently, with the goal of improving generalisation for tail classes. Most existing works use supervised learning without considering the prevailing noise in the training dataset. To move long-tailed learning towards more realistic scenarios, this work investigates the label noise problem under long-tailed label distribution. We first observe the negative impact of noisy labels on the performance of existing methods, revealing the intrinsic challenges of this problem. As the most commonly used approach to cope with noisy labels in previous literature, we then find that the small-loss trick fails under long-tailed label distribution. The reason is that deep neural networks cannot distinguish correctly-labeled and mislabeled examples on tail classes. To overcome this limitation, we establish a new prototypical noise detection method by designing a distance-based metric that is resistant to label noise. Based on the above findings, we propose a robust framework,~\algo, that realizes noise detection for long-tailed learning, followed by soft pseudo-labeling via both label smoothing and diverse label guessing. Moreover, our framework can naturally leverage semi-supervised learning algorithms to further improve the generalisation. Extensive experiments on benchmark and real-world datasets demonstrate the superiority of our methods over existing baselines. In particular, our method outperforms DivideMix by 3\% in test accuracy. Source code will be released soon.

Motivation & Objective

• Motivate long-tailed learning in the presence of label noise to reflect realistic data conditions.
• Show that small-loss-based noise detection fails under long-tailed distributions and propose a geometry-based alternative.
• Develop RoLT to detect noisy samples via a prototypical distance detector and to perform soft pseudo-labeling.
• Leverage semi-supervised learning ideas to further improve generalisation with clean and noisy subsets.
• Demonstrate robustness and superiority over baselines on benchmark and real-world datasets.

Proposed method

• Compute per-class prototypes as normalized averages of embeddings to capture class geometry.
• Model distances to prototypes with a two-component Gaussian mixture to separate clean and noisy data in a class-independent way.
• Refine prototypes using the subset of detected clean examples to improve noise detection.
• Generate soft pseudo-labels for noisy samples by combining predictions from ERM and NCM classifiers and original labels, with label smoothing and temporal ensembling.
• Train with a combined loss on clean data and softly labeled noisy data, enabling semi-supervised-style learning within RoLT.

Experimental results

Research questions

• RQ1 Can a class-independent, prototypical distance detector robustly separate clean and noisy data under long-tailed class distributions?
• RQ2 Does soft pseudo-labeling that blends ERM, NCM predictions, and label smoothing improve learning with noisy tail data?
• RQ3 How does RoLT perform compared with standard long-tailed methods and existing noisy-label approaches on benchmark and real-world noisy datasets?
• RQ4 Can RoLT be extended with semi-supervised learning techniques to further boost generalisation?

Key findings

• RoLT significantly outperforms ERM and several long-tailed baselines across CIFAR-10/100 under varying imbalance ratios and noise levels.
• RoLT–DRW and RoLT+ achieve larger gains as noise increases, highlighting effective noise detection and robust learning.
• NCM-based predictions provide more balanced tail-class recall than ERM, supporting the proposed prototypical noise detection strategy.
• Soft labeling via label smoothing and diverse label guessing improves performance over using noisy labels or ERM predictions alone.
• OnWebVision, RoLT+ achieves competitive top-1/top-5 accuracy compared to state-of-the-art noisy-label and long-tailed methods.