[Paper Review] Curriculum Loss: Robust Learning and Generalization against Label Corruption

Yueming Lyu, Ivor W. Tsang|arXiv (Cornell University)|May 24, 2019

Machine Learning and Data Classification29 references83 citations

TL;DR

Proposes Curriculum Loss (CL) as a tight upper bound to the 0-1 loss that adaptively selects training samples, improving robustness to label noise. Extends to Noise Pruned Curriculum Loss (NPCL) for higher noise rates and supports mini-batch training.

ABSTRACT

Deep neural networks (DNNs) have great expressive power, which can even memorize samples with wrong labels. It is vitally important to reiterate robustness and generalization in DNNs against label corruption. To this end, this paper studies the 0-1 loss, which has a monotonic relationship with an empirical adversary (reweighted) risk~\citep{hu2016does}. Although the 0-1 loss has some robust properties, it is difficult to optimize. To efficiently optimize the 0-1 loss while keeping its robust properties, we propose a very simple and efficient loss, i.e. curriculum loss (CL). Our CL is a tighter upper bound of the 0-1 loss compared with conventional summation based surrogate losses. Moreover, CL can adaptively select samples for model training. As a result, our loss can be deemed as a novel perspective of curriculum sample selection strategy, which bridges a connection between curriculum learning and robust learning. Experimental results on benchmark datasets validate the robustness of the proposed loss.

Motivation & Objective

• Motivate robustness and generalization of deep nets under label corruption and noisy labels.
• Introduce a tighter upper bound of the 0-1 loss that remains efficient to optimize.
• Develop an adaptive curriculum-based sample selection mechanism for training.
• Extend CL with Noise Pruning to handle high noise rates.
• Provide practical, plug-in loss functions for deep models with mini-batch updates.

Proposed method

• Define Curriculum Loss (CL) as a tighter upper bound of the 0-1 loss using a base loss l(u) with l(u) ≥ 1(u<0).
• Show CL bounds J(u) ≤ Q(u) ≤ ŴJ(u) and enable adaptive sample selection via a partial optimization over a binary selector v.
• Provide an O(n log n) algorithm (Algorithm 1) to compute the optimal sample subset for CL with mini-batch support.
• Introduce Noise Pruned Curriculum Loss (NPCL) by incorporating a prior on the noise rate ε and a pruning mechanism (Eq. 18) that removes likely noisy samples during training.
• Present batch-based versions (шL) for efficient mini-batch training (Eq. 20–21).
• Describe training procedures (Algorithm 2) that integrate sample selection into standard deep learning workflows.

Experimental results

Research questions

• RQ1 Can a tighter upper bound of the 0-1 loss improve robustness to label corruption without sacrificing optimization efficiency?
• RQ2 Does adaptive curriculum-like sample selection improve generalization under noisy labels beyond existing robust losses?
• RQ3 How does Noise Pruned Curriculum Loss (NPCL) perform relative to baseline robust losses across varying noise rates and datasets?
• RQ4 Can the proposed losses be implemented as plug-ins with mini-batch updates in common deep learning frameworks?
• RQ5 What are the computational properties (complexity, convergence) of the sample-selection algorithms embedded in CL/NPCL?

Key findings

• CL provides a tighter upper bound on the 0-1 loss than conventional surrogate losses, enabling robust learning.
• A simple O(n log n) algorithm (Algorithm 1) can select samples adaptively for curriculum training with mini-batch support.
• NPCL improves robustness at higher noise rates by pruning likely noisy samples using a prior ε and the curriculum framework.
• Empirical results show NPCL often outperforms Generalized Cross-Entropy (GCE) and matches or exceeds Co-teaching methods in several settings, especially under stronger noise.
• The approach works as a plug-in loss for deep models and remains computationally efficient with batch updates.
• Various experiments on MNIST, CIFAR-10/100, and Tiny-ImageNet demonstrate robustness across symmetric and pairwise label corruption scenarios.