[Paper Review] Self-Adaptive Training: beyond Empirical Risk Minimization

Lang Huang, Chao Zhang+1 more|arXiv (Cornell University)|Feb 24, 2020

Machine Learning and Data ClassificationComputer Science54 references88 citations

TL;DR

The paper introduces Self-Adaptive Training, a method that uses accumulated model predictions to adapt training targets and sample weights, improving robustness to label and input noise without extra computation. It outperforms ERM across noisy and adversarial settings and enables improvements in label-noise classification and selective classification.

ABSTRACT

We propose self-adaptive training---a new training algorithm that dynamically corrects problematic training labels by model predictions without incurring extra computational cost---to improve generalization of deep learning for potentially corrupted training data. This problem is crucial towards robustly learning from data that are corrupted by, e.g., label noises and out-of-distribution samples. The standard empirical risk minimization (ERM) for such data, however, may easily overfit noises and thus suffers from sub-optimal performance. In this paper, we observe that model predictions can substantially benefit the training process: self-adaptive training significantly improves generalization over ERM under various levels of noises, and mitigates the overfitting issue in both natural and adversarial training. We evaluate the error-capacity curve of self-adaptive training: the test error is monotonously decreasing w.r.t. model capacity. This is in sharp contrast to the recently-discovered double-descent phenomenon in ERM which might be a result of overfitting of noises. Experiments on CIFAR and ImageNet datasets verify the effectiveness of our approach in two applications: classification with label noise and selective classification. We release our code at https://github.com/LayneH/self-adaptive-training.

Motivation & Objective

• Motivate robust learning when training data are corrupted by random or adversarial noise beyond standard ERM.
• Propose a training mechanism that leverages model predictions to guide training dynamics.
• Demonstrate improved generalization under label noise and adversarial attacks on CIFAR/ImageNet.
• Show applicability to selective classification and analyze robustness and efficiency.

Proposed method

• Introduce an exponential moving average of model predictions to progressively correct training targets.
• Compute sample weights as the maximum entry of the updated target to reflect labeling confidence.
• Train with a re-weighted cross-entropy loss that normalizes by total sample weights.
• Maintain compatibility with existing architectures and training pipelines with near-zero extra cost.

Experimental results

Research questions

• RQ1 Can model predictions be used dynamically to correct training signals to mitigate overfitting to noisy data?
• RQ2 Does self-adaptive training reduce generalization error under random and adversarial noise compared to ERM?
• RQ3 Can the approach improve performance in applications like classification with label noise and selective classification?

Key findings

• Self-adaptive training mitigates overfitting on noisy data and yields lower generalization error than ERM across multiple noise types and levels.
• Under random noise, the method yields a single-descent error-capacity curve, unlike the double-descent seen in ERM.
• The approach improves adversarial robustness by 1–3% over TRADES under white-box PGD attacks.
• On CIFAR-10/100 with label noise, it achieves up to 9.3 percentage points higher accuracy than prior methods, and on ImageNet it improves ERM by about 2% under 40% label noise.
• For selective classification, it achieves up to 50% relative improvement over state-of-the-art across datasets and coverage levels.