[Paper Review] Push the Student to Learn Right: Progressive Gradient Correcting by Meta-learner on Corrupted Labels.

Jun Shu, Qi Xie+5 more|arXiv (Cornell University)|Feb 20, 2019

Machine Learning and Data Classification33 references4 citations

TL;DR

This paper proposes a simple yet effective meta-learning method for robust learning under label corruption by progressively correcting classifier gradients using a lightweight meta-learner with minimal hyperparameter tuning. Guided by a small set of unbiased meta-data, the method adaptively adjusts sample weights to correct learning bias, demonstrating superior robustness, stability, and efficiency on corrupted label benchmarks.

ABSTRACT

While deep networks have strong fitting capability to complex input patterns, they can easily overfit to biased training data with corrupted labels. Sample reweighting strategy is commonly used to alleviate this robust learning issue, through imposing zero or possibly smaller weights to corrupted samples to suppress their negative influence to learning. Current reweighting algorithms, however, need elaborate tuning of additional hyper-parameters or careful designing of complex meta-learner for learning to assign weights on samples. To address these issues, we propose a new meta-learning method with few tuned hyper-parameters and simple structure of a meta-learner (one hidden layer MLP network). Guided by a small amount of unbiased meta-data, the parameters of the proposed meta-learner can be gradually evolved for finely tugging the classifier gradient approaching to the right direction. This learning manner complies with a real teaching progress: A good teacher should more respect the student's own learning manner and help progressively correct his learning bias based on his/her current learning status. Experimental results substantiate the robustness of the new algorithm on corrupted label cases, as well as its stability and efficiency in learning.

Motivation & Objective

• To address the challenge of deep neural networks overfitting to biased or corrupted training labels.
• To reduce reliance on complex meta-learner designs and extensive hyperparameter tuning in existing sample reweighting methods.
• To develop a simple, stable, and efficient meta-learning framework that progressively corrects learning bias based on the student's current learning status.
• To enable effective gradient correction by guiding the classifier toward the correct learning direction using minimal unbiased meta-data.

Proposed method

• A meta-learner with a single hidden layer MLP is trained to predict sample weights based on model activations and gradients.
• The meta-learner is optimized using a small set of unbiased meta-data to guide the classifier's gradient updates.
• The method progressively corrects the classifier's gradient by adjusting sample weights in a way that mimics a teacher guiding a student's learning process.
• The meta-learner's parameters are updated iteratively to refine weight assignments, minimizing interference from corrupted samples.
• The approach avoids complex loss design or extensive hyperparameter tuning, relying instead on a simple, scalable meta-learning loop.

Experimental results

Research questions

• RQ1 Can a simple meta-learner with minimal hyperparameters effectively correct gradient updates in the presence of label corruption?
• RQ2 How does progressive gradient correction based on current learning status improve robustness compared to static or complex reweighting methods?
• RQ3 To what extent does the method maintain stability and efficiency across different levels of label noise?
• RQ4 How does the use of a small unbiased meta-data set impact the performance of the meta-learner in correcting learning bias?

Key findings

• The proposed method achieves superior robustness on datasets with corrupted labels, outperforming existing reweighting baselines in accuracy under high noise levels.
• The method demonstrates high stability across multiple random seeds and noise settings, indicating consistent performance.
• The model maintains high efficiency due to the simple one-hidden-layer meta-learner and minimal hyperparameter tuning.
• The progressive correction mechanism effectively reduces overfitting to corrupted samples by dynamically adjusting sample weights based on learning status.