[Paper Review] A Survey of Label-noise Representation Learning: Past, Present and Future

Bo Han, Quanming Yao+5 more|arXiv (Cornell University)|Nov 9, 2020

Machine Learning and Data ClassificationComputer Science149 references101 citations

TL;DR

A comprehensive survey defining Label-noise Representation Learning (LNRL), surveying theory, taxonomy, and methods to robustly train deep models under noisy labels, and outlining future directions.

ABSTRACT

Classical machine learning implicitly assumes that labels of the training data are sampled from a clean distribution, which can be too restrictive for real-world scenarios. However, statistical-learning-based methods may not train deep learning models robustly with these noisy labels. Therefore, it is urgent to design Label-Noise Representation Learning (LNRL) methods for robustly training deep models with noisy labels. To fully understand LNRL, we conduct a survey study. We first clarify a formal definition for LNRL from the perspective of machine learning. Then, via the lens of learning theory and empirical study, we figure out why noisy labels affect deep models' performance. Based on the theoretical guidance, we categorize different LNRL methods into three directions. Under this unified taxonomy, we provide a thorough discussion of the pros and cons of different categories. More importantly, we summarize the essential components of robust LNRL, which can spark new directions. Lastly, we propose possible research directions within LNRL, such as new datasets, instance-dependent LNRL, and adversarial LNRL. We also envision potential directions beyond LNRL, such as learning with feature-noise, preference-noise, domain-noise, similarity-noise, graph-noise and demonstration-noise.

Motivation & Objective

• Define Label-noise Representation Learning (LNRL) and its scope.
• Explain why noisy labels affect deep models through learning-theory and empirical perspectives.
• Provide a unified taxonomy of LNRL methods based on data, objective, and optimization.
• Survey existing approaches that use noise transition matrices, loss correction, and optimization tricks.
• Propose future research directions and datasets for LNRL beyond label noise.

Proposed method

• Formalize LNRL with a general problem setup where training labels are corrupted.
• Survey theoretical foundations on data, objective, and optimization perspectives.
• Develop a unified taxonomy of methods along data (noise transition matrix), objective (noise-tolerant losses), and optimization (memorization-based strategies).
• Discuss anchor points, transition matrices, and loss correction as core tools.
• Highlight memorization effects and early stopping as optimization guidance.
• Summarize future directions and potential directions beyond LNRL.

Experimental results

Research questions

• RQ1 What is the formal definition and scope of Label-noise Representation Learning (LNRL)?
• RQ2 Why do noisy labels impact deep models, from learning-theory and empirical perspectives?
• RQ3 How can LNRL methods be categorized and what are the pros/cons of each category?
• RQ4 What are essential components and future directions for robust LNRL, including datasets and adversarial settings?

Key findings

• LNRL integrates data, objective, and optimization to robustly learn with noisy labels.
• Estimating and utilizing the label noise transition matrix is central to many approaches.
• Noise-tolerant losses and classifier-consistent estimators help bridge noisy and clean distributions.
• Optimization policies leveraging memorization effects and early stopping can improve robustness.
• A unified taxonomy clarifies strengths and trade-offs of different LNRL strategies.
• Future directions include instance-dependent noise, adversarial LNRL, and learning with various noise modalities.