[Paper Review] Deep learning with noisy labels: exploring techniques and remedies in medical image analysis

Davood Karimi, Haoran Dou+2 more|arXiv (Cornell University)|Dec 5, 2019

Machine Learning and Data Classification4 citations

TL;DR

This paper investigates label noise in medical image datasets—arising from inter-observer variability, human error, and automated labeling—and evaluates techniques to mitigate its impact on deep learning models. By experimenting on three medical imaging datasets, the authors demonstrate that modeling annotator confusion and leveraging uncertainty-aware training significantly improve performance over simple label filtering or ignoring high-loss samples.

ABSTRACT

Supervised training of deep learning models requires large labeled datasets. There is a growing interest in obtaining such datasets for medical image analysis applications. However, the impact of label noise has not received sufficient attention. Recent studies have shown that label noise can significantly impact the performance of deep learning models in many machine learning and computer vision applications. This is especially concerning for medical applications, where datasets are typically small, labeling requires domain expertise and suffers from high inter- and intra-observer variability, and erroneous predictions may influence decisions that directly impact human health. In this paper, we first review the state-of-the-art in handling label noise in deep learning. Then, we review studies that have dealt with label noise in deep learning for medical image analysis. Our review shows that recent progress on handling label noise in deep learning has gone largely unnoticed by the medical image analysis community. To help achieve a better understanding of the extent of the problem and its potential remedies, we conducted experiments with three medical imaging datasets with different types of label noise, where we investigated several existing strategies and developed new methods to combat the negative effect of label noise. Based on the results of these experiments and our review of the literature, we have made recommendations on methods that can be used to alleviate the effects of different types of label noise on deep models trained for medical image analysis. We hope that this article helps the medical image analysis researchers and developers in choosing and devising new techniques that effectively handle label noise in deep learning.

Motivation & Objective

• To investigate the prevalence and impact of label noise in medical image datasets, particularly due to inter-observer variability and automated labeling.
• To evaluate existing and novel deep learning techniques for handling label noise in medical image analysis applications.
• To provide evidence-based recommendations for selecting and applying noise-robust training strategies in clinical deep learning pipelines.
• To demonstrate that modeling annotator uncertainty and confusion matrices leads to better performance than label filtering or ignoring high-loss samples.

Proposed method

• Evaluated multiple label noise mitigation strategies, including loss correction, label smoothing, and uncertainty-aware training.
• Proposed a method to detect and correct missing annotations by leveraging CNN prediction uncertainty to identify under-annotated lesions.
• Applied a confusion matrix-based approach to model individual expert annotation errors during end-to-end training.
• Used computer-generated noisy labels from related imaging sequences (e.g., T2-weighted to DW-MRI) as weak supervision to improve model generalization.
• Compared performance across methods including label filtering, loss weighting, and ensemble-based label fusion.
• Trained models on datasets with varying noise types: expert-annotated lesions with missing small structures, multi-expert segmentation with inter-observer variability, and atlas-based noisy segmentations.

Experimental results

Research questions

• RQ1 How does label noise from inter-observer variability affect deep learning model performance in medical image analysis?
• RQ2 Which label noise mitigation techniques are most effective for different types of label noise in medical imaging?
• RQ3 Can modeling annotator confusion during training outperform traditional label fusion or filtering strategies?
• RQ4 To what extent can weakly supervised labels from related imaging modalities improve model performance when ground truth is scarce?
• RQ5 Does uncertainty-based detection of missing annotations lead to improved generalization in lesion segmentation tasks?

Key findings

• Modeling annotator confusion during training significantly outperformed label fusion and loss filtering in multi-expert segmentation tasks.
• The use of CNN prediction uncertainty enabled effective detection and correction of missing annotations in small, faint, or isolated lesions.
• Training with computer-generated noisy labels from matched T2-weighted images improved model performance over training with only 65 manually segmented DW-MRI volumes.
• Ignoring high-loss samples was less effective than modeling label uncertainty, particularly in datasets with systematic annotation biases.
• Label noise from automated systems was harmful if too inaccurate, but useful when sufficiently aligned with true anatomy.
• Multi-expert labeling with confusion modeling led to higher performance gains than single-expert labeling or simple averaging.