[Paper Review] Normalized Loss Functions for Deep Learning with Noisy Labels

Xingjun Ma, Hanxun Huang+4 more|arXiv (Cornell University)|Jun 24, 2020

Machine Learning and Data Classification125 citations

TL;DR

The paper shows that any loss can be made robust to noisy labels via normalization, introduces Active Passive Loss (APL) to address underfitting, and demonstrates state-of-the-art performance on benchmarks under heavy label noise.

ABSTRACT

Robust loss functions are essential for training accurate deep neural networks (DNNs) in the presence of noisy (incorrect) labels. It has been shown that the commonly used Cross Entropy (CE) loss is not robust to noisy labels. Whilst new loss functions have been designed, they are only partially robust. In this paper, we theoretically show by applying a simple normalization that: any loss can be made robust to noisy labels. However, in practice, simply being robust is not sufficient for a loss function to train accurate DNNs. By investigating several robust loss functions, we find that they suffer from a problem of underfitting. To address this, we propose a framework to build robust loss functions called Active Passive Loss (APL). APL combines two robust loss functions that mutually boost each other. Experiments on benchmark datasets demonstrate that the family of new loss functions created by our APL framework can consistently outperform state-of-the-art methods by large margins, especially under large noise rates such as 60% or 80% incorrect labels.

Motivation & Objective

• Motivate robust learning with noisy labels and compare existing robust losses.
• Propose a normalization technique to make any loss robust to label noise.
• Identify underfitting in robust losses and design the Active Passive Loss (APL) framework.
• Empirically demonstrate robustness and improved learning with APL on benchmark datasets under high noise rates.

Proposed method

• Normalize a loss by L_norm = L(f(x), y) / sum_j L(f(x), j) to achieve robustness to label noise.
• Define and compare active versus passive losses; active maximizes p(y|x) while passive also minimizes other class probabilities.
• Propose Active Passive Loss (APL) as L_APL = alpha * L_Active + beta * L_Passive and prove robustness is preserved under certain conditions.
• Classify existing robust losses (e.g., CE, MAE, RCE, NCE, NFL) as active or passive and show how APL combines them to address underfitting.
• Provide theoretical lemmas showing noise-tolerance under symmetric and asymmetric noise with normalized losses.
• Conduct empirical studies on MNIST, CIFAR-10/100, and WebVision to compare APL against state-of-the-art methods.

Experimental results

Research questions

• RQ1 Can normalization render any loss robust to noisy labels in multi-class classification?
• RQ2 Why do some robust losses underfit, and can a combined active-passive framework improve both robustness and learning?
• RQ3 Does the Active Passive Loss (APL) framework achieve state-of-the-art robustness and learning efficiency under high noise rates?
• RQ4 How should the balance between active and passive terms be set across simple and complex datasets?
• RQ5 Do normalized variants of existing losses maintain robustness while enabling effective learning?

Key findings

• Normalized losses are robust to symmetric and asymmetric label noise under specified rate conditions.
• Simple normalization preserves robustness but robustness alone is not sufficient for high accuracy due to underfitting.
• APL combining an active loss and a passive loss yields robustness plus improved learning efficiency, outperforming state-of-the-art methods at high noise (60–80%).
• Scaling normalized losses does not consistently resolve underfitting; proper active-passive balance is crucial.
• On CIFAR-10/100 and MNIST, APL variants (e.g., NCE+RCE, NFL+RCE) achieve competitive or superior performance across noise settings compared to GCE, NLNL, SCE.