[Paper Review] Open-set Label Noise Can Improve Robustness Against Inherent Label Noise

Hongxin Wei, Lue Tao+2 more|arXiv (Cornell University)|Jun 21, 2021

Machine Learning and Data Classification77 references30 citations

TL;DR

This paper shows that open-set label noise can be non-toxic or even beneficial for robustness to inherent label noise, and introduces Open-set Regularization with Dynamic Noisy Labels (ODNL) to consume model capacity and improve generalization and OOD detection.

ABSTRACT

Learning with noisy labels is a practically challenging problem in weakly supervised learning. In the existing literature, open-set noises are always considered to be poisonous for generalization, similar to closed-set noises. In this paper, we empirically show that open-set noisy labels can be non-toxic and even benefit the robustness against inherent noisy labels. Inspired by the observations, we propose a simple yet effective regularization by introducing Open-set samples with Dynamic Noisy Labels (ODNL) into training. With ODNL, the extra capacity of the neural network can be largely consumed in a way that does not interfere with learning patterns from clean data. Through the lens of SGD noise, we show that the noises induced by our method are random-direction, conflict-free and biased, which may help the model converge to a flat minimum with superior stability and enforce the model to produce conservative predictions on Out-of-Distribution instances. Extensive experimental results on benchmark datasets with various types of noisy labels demonstrate that the proposed method not only enhances the performance of many existing robust algorithms but also achieves significant improvement on Out-of-Distribution detection tasks even in the label noise setting.

Motivation & Objective

• Investigate whether open-set noisy labels can be harmless or beneficial for generalization under label noise.
• Analyze how open-set auxiliary data affects training dynamics and robustness to inherent label noise.
• Propose a practical regularization technique (ODNL) that uses dynamic open-set labels to consume neural network capacity without harming learning from clean data.
• Theoretically relate the method to SGD noise and contrast with related approaches (OE, SLN, label randomization).
• Demonstrate effectiveness across CIFAR-10/100 and Clothing1M, including improvements to OOD detection under label noise.

Proposed method

• Introduce Open-set samples with Dynamic Noisy Labels (ODNL) by pairing open-set auxiliary data with randomly and dynamically assigned labels drawn uniformly from the label set during training.
• Formulate the training objective as L_total = L1 + eta * L2, where L1 is standard cross-entropy on the clean data and L2 enforces consistency with dynamic random labels on open-set data.
• Provide SGD-noise analysis showing ODNL induces random-direction, conflict-free, and biased noise that helps converge to flat minima and promotes conservative OOD predictions.
• Show that ODNL can be integrated with existing robust training methods (sample selection, loss correction, robust losses) to boost their performance.
• Compare ODNL to related techniques (OE, SLN, DisturbLabel) and discuss differences in dynamism and impact on robustness to inherent label noise.

Experimental results

Research questions

• RQ1 Can open-set noisy labels be harmless or beneficial for generalization under inherent label noise?
• RQ2 How does introducing open-set auxiliary data with dynamic noisy labels affect training dynamics and robustness to label noise?
• RQ3 Can ODNL improve OOD detection and be combined with existing robust training methods?
• RQ4 What is the relationship between ODNL-induced SGD noise and other label-noise regularization approaches?
• RQ5 Is ODNL effective on standard benchmarks with real-world noisy labels (CIFAR-10/100, Clothing1M)?

Key findings

• Open-set noisy labels can be harmless or beneficial for robustness to inherent label noise when the auxiliary dataset size is large enough.
• ODNL consistently improves robustness across CIFAR-10/100 and Clothing1M, and enhances OOD detection under label noise.
• ODNL enhances performance of various existing robust training methods (e.g., Decoupling, F-correction, PHuber-CE, Co-teaching, JoCoR).
• SGD-noise analysis shows ODNL introduces random-direction, conflict-free, and biased noise that helps reach flat minima and yields conservative OOD predictions.
• Compared to Outlier Exposure (OE), ODNL provides dynamic noise and shows superior improvement in robustness to noisy labels.