[Paper Review] Joint Optimization Framework for Learning with Noisy Labels

Daiki Tanaka, Daiki Ikami+2 more|arXiv (Cornell University)|Mar 30, 2018

Machine Learning and Data Classification4 references105 citations

TL;DR

This paper proposes a joint optimization of CNN parameters and true labels to learn from noisy labeled data, using alternating updates and soft-labels to outperform state-of-the-art methods on CIFAR-10 with noise and Clothing1M.

ABSTRACT

Deep neural networks (DNNs) trained on large-scale datasets have exhibited significant performance in image classification. Many large-scale datasets are collected from websites, however they tend to contain inaccurate labels that are termed as noisy labels. Training on such noisy labeled datasets causes performance degradation because DNNs easily overfit to noisy labels. To overcome this problem, we propose a joint optimization framework of learning DNN parameters and estimating true labels. Our framework can correct labels during training by alternating update of network parameters and labels. We conduct experiments on the noisy CIFAR-10 datasets and the Clothing1M dataset. The results indicate that our approach significantly outperforms other state-of-the-art methods.

Motivation & Objective

• Motivation to train DNNs on large-scale datasets that contain noisy labels from web sources.
• Develop a framework that avoids memorization of noisy labels by jointly optimizing parameters and labels.
• Demonstrate that high learning rates help prevent memorization of noise and improve robustness.
• Evaluate the proposed framework on synthetic (SN-CIFAR, AN-CIFAR, PL-CIFAR) and real (Clothing1M) noisy datasets.

Proposed method

• Define a joint objective that optimizes network parameters θ and label matrix Y to minimize L(θ, Y | X) comprising three terms: Lc (KL-divergence between labels and predictions), Lp (regularization enforcing a class prior p), and Le (entropy-based regularization).
• Use alternating optimization: update θ with SGD on L(θ, Y|X) while keeping Y fixed, then update Y per sample using either hard or soft labels, with soft labels performing better.
• Adopt soft-label updates y_i = s(θ, x_i) and use a prior p (uniform for CIFAR-10 in experiments) to avoid trivial solutions.
• Exploit high learning rates to reduce memorization of noisy labels and observe different memorization dynamics depending on lr in experiments.
• Incorporate an entropy term Le to keep soft-label distributions sharp enough to concentrate on single classes.
• Conduct a two-step training process: first update labels and then train the network on the obtained labels.

Experimental results

Research questions

• RQ1 Can joint optimization of network parameters and labels mitigate the memorization of noisy labels in DNNs?
• RQ2 Does using soft-label updates outperform hard-label updates in recovering clean labels and improving test accuracy?
• RQ3 How do learning rate choices influence memorization of noisy labels and overall performance under noisy supervision?
• RQ4 How does the proposed method perform on synthetic noisy CIFAR-10 variants and real-world Clothing1M data?
• RQ5 What is the impact of regularization terms Lp and Le on preventing degenerate label assignments and promoting robust learning?

Key findings

• The proposed joint optimization framework yields state-of-the-art results on SN-CIFAR and competitive performance on Clothing1M compared to prior methods.
• Soft-label updates outperform hard-label updates, enabling faster convergence and higher test accuracy by leveraging prediction confidences.
• A high learning rate helps prevent memorization of noisy labels, delaying or avoiding fitting to noise during training.
• The two-step training process (label optimization followed by standard supervised training) improves robustness to label noise.
• Regularization using a class-prior KL term and an entropy term stabilizes label updates and prevents collapse to a single class.