[Paper Review] Robust Loss Functions under Label Noise for Deep Neural Networks

Aritra Ghosh, Himanshu Kumar+1 more|arXiv (Cornell University)|Dec 27, 2017

Machine Learning and Data ClassificationComputer Science8 references151 citations

TL;DR

The paper derives sufficient conditions for loss functions to be noise-tolerant in multiclass settings and shows mean absolute error (MAE) is inherently robust to label noise in deep networks, with theoretical results and empirical validation.

ABSTRACT

In many applications of classifier learning, training data suffers from label noise. Deep networks are learned using huge training data where the problem of noisy labels is particularly relevant. The current techniques proposed for learning deep networks under label noise focus on modifying the network architecture and on algorithms for estimating true labels from noisy labels. An alternate approach would be to look for loss functions that are inherently noise-tolerant. For binary classification there exist theoretical results on loss functions that are robust to label noise. In this paper, we provide some sufficient conditions on a loss function so that risk minimization under that loss function would be inherently tolerant to label noise for multiclass classification problems. These results generalize the existing results on noise-tolerant loss functions for binary classification. We study some of the widely used loss functions in deep networks and show that the loss function based on mean absolute value of error is inherently robust to label noise. Thus standard back propagation is enough to learn the true classifier even under label noise. Through experiments, we illustrate the robustness of risk minimization with such loss functions for learning neural networks.

Motivation & Objective

• Provide analytical conditions under which a loss function is robust to label noise in multiclass classification.
• Generalize binary-noise robustness results to multiclass settings.
• Identify commonly used loss functions that satisfy noise-tolerance, with emphasis on MAE.
• Demonstrate empirically the robustness of these loss functions in deep networks.

Proposed method

• Define L-risk and noise models (symmetric/uniform, simple non-uniform, class-conditional).
• Establish sufficient conditions for a loss to be noise-tolerant (symmetry condition and related bounds).
• Prove robustness results (Theorems 1–3) for symmetric, simple non-uniform, and class-conditional noise.
• Show that MAE satisfies the symmetry condition and is thus robust to label noise.
• Compare MAE, MSE, and CCE experimentally on image and text datasets under various noise regimes.

Experimental results

Research questions

• RQ1 Under what conditions is a loss function robust to label noise in multiclass classification?
• RQ2 Which common loss functions satisfy these conditions, and how do they perform under noise?
• RQ3 Can MAE offer inherent robustness in deep networks without modifying architecture or estimation procedures?
• RQ4 What is the empirical robustness of MAE compared to CCE and MSE across datasets and noise types?

Key findings

• MAE satisfies the proposed symmetry condition, rendering it noise-tolerant under symmetric label noise in multiclass settings.
• Symmetric losses are robust to uniform label noise if the noise rate satisfies eta < (k-1)/k.
• For simple non-uniform noise, if the noise-free risk is zero, the loss remains robust; otherwise a bound on the noisy risk is derived.
• Under class-conditional noise, robustness holds when the noise matrix is diagonally dominant and the loss is bounded as specified.
• Empirical results show MAE often maintains higher test accuracy than CCE under symmetric and class-conditional label noise across MNIST, RCV1, CIFAR-10, IMDB, and other datasets.
• MSE provides intermediate robustness due to its boundedness, but is less robust than MAE under heavy label noise.