[Paper Review] Learning Adaptive Loss for Robust Learning with Noisy Labels

Jun Shu, Qian Zhao+3 more|arXiv (Cornell University)|Feb 16, 2020

Machine Learning and Data Classification56 references15 citations

TL;DR

This paper proposes an adaptive robust learning (ARL) framework that jointly optimizes network weights and hyperparameters of robust loss functions via meta-learning, enabling automatic, data-driven hyperparameter tuning. The method consistently outperforms conventional hyperparameter tuning and state-of-the-art baselines across multiple robust loss functions and datasets, demonstrating superior generalization under label noise.

ABSTRACT

Robust loss minimization is an important strategy for handling robust learning issue on noisy labels. Current robust loss functions, however, inevitably involve hyperparameter(s) to be tuned, manually or heuristically through cross validation, which makes them fairly hard to be generally applied in practice. Besides, the non-convexity brought by the loss as well as the complicated network architecture makes it easily trapped into an unexpected solution with poor generalization capability. To address above issues, we propose a meta-learning method capable of adaptively learning hyperparameter in robust loss functions. Specifically, through mutual amelioration between robust loss hyperparameter and network parameters in our method, both of them can be simultaneously finely learned and coordinated to attain solutions with good generalization capability. Four kinds of SOTA robust loss functions are attempted to be integrated into our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its accuracy and generalization performance, as compared with conventional hyperparameter tuning strategy, even with carefully tuned hyperparameters.

Motivation & Objective

• To address the challenge of manually tuning hyperparameters in robust loss functions, which limits practical deployment and generalization.
• To reduce reliance on cross-validation and heuristic tuning by enabling end-to-end, adaptive learning of robust loss hyperparameters.
• To improve generalization in deep learning under label noise by jointly optimizing network parameters and robust loss hyperparameters.
• To validate the generality and effectiveness of the proposed method across diverse state-of-the-art robust loss functions.
• To provide a theoretically grounded, iterative meta-learning framework that enhances solution quality in highly non-convex optimization settings.

Proposed method

• Proposes a meta-learning framework that alternately updates network parameters and robust loss hyperparameters using a bi-level optimization scheme.
• Introduces a mutual amelioration mechanism where hyperparameters and network weights are refined in tandem to improve generalization.
• Applies the framework to four SOTA robust loss functions: GCE, SL, Bi-Tempered, and PolySoft, demonstrating broad applicability.
• Employs a two-stage training process: inner loop optimizes network weights for a given hyperparameter, outer loop updates the hyperparameter using meta-validation loss.
• Uses SGD with adaptive learning rates for both network and hyperparameter optimization, with learning rate decay and weight decay for stability.
• The method is designed to reduce overfitting to noisy labels by dynamically adjusting the robustness of the loss function during training.

Experimental results

Research questions

• RQ1 Can a meta-learning approach effectively and automatically tune hyperparameters in robust loss functions without manual cross-validation?
• RQ2 Does the joint optimization of network weights and robust loss hyperparameters lead to better generalization than fixed hyperparameter settings?
• RQ3 How does the proposed adaptive learning strategy compare to conventional hyperparameter tuning in terms of accuracy and robustness under varying noise rates?
• RQ4 Can the framework be generalized across different types of SOTA robust loss functions, including those with theoretical robustness guarantees?
• RQ5 Does the iterative mutual amelioration between hyperparameters and network parameters yield better solutions than sequential optimization?

Key findings

• The ARL method consistently outperforms conventional hyperparameter tuning with cross-validation across all tested robust loss functions, including GCE, SL, Bi-Tempered, and PolySoft.
• On CIFAR-10 with varying noise rates, ARL achieved higher test accuracy than baseline methods, with improvements of up to 5.2% over standard SL under 40% noise.
• On the real-world Clothing1M dataset, A-PolySoft achieved the highest top-1 accuracy of 87.6%, surpassing all baselines and demonstrating strong generalization on large-scale noisy data.
• Ablation studies confirmed that the mutual amelioration between hyperparameters and network weights is essential, as ARL outperformed strategies that fix one while optimizing the other.
• t-SNE visualization showed that representations learned by ARL-based models (e.g., A-Bi-Tempered) were more separable and better clustered than those from CE or standard Bi-Tempered.
• The method proved effective even when hyperparameters were not optimally tuned, indicating robustness to initialization and noise.