[Paper Review] The Impact of an Inter-rater Bias on Neural Network Training

Or Shwartzman, Harel Gazit+2 more|arXiv (Cornell University)|Jun 12, 2019

AI in cancer detection11 references4 citations

TL;DR

This paper investigates how inter-rater bias—differences in manual segmentation annotations between raters—impacts deep neural network (DNN) training for medical image segmentation. Using the ISBI 2015 MS challenge dataset, it demonstrates that DNNs trained on annotations from one rater produce lower Dice scores and biased lesion load estimates when tested on data from another rater, revealing that model performance is sensitive to rater-specific biases rather than just random variability.

ABSTRACT

The problem of inter-rater variability is often discussed in the context of manual labeling of medical images. It is assumed to be bypassed by automatic model-based approaches for image segmentation which are considered `objective', providing single, deterministic solutions. However, the emergence of data-driven approaches such as Deep Neural Networks (DNNs) and their application to supervised semantic segmentation - brought this issue of raters' disagreement back to the front-stage. In this paper, we highlight the issue of inter-rater bias as opposed to random inter-observer variability and demonstrate its influence on DNN training, leading to different segmentation results for the same input images. In fact, lower Dice scores are calculated if training and test segmentations are of different raters. Moreover, we demonstrate that inter-rater bias in the training examples is amplified when considering the segmentation predictions for the test data. We support our findings by showing that a classifier-DNN trained to distinguish between raters based on their manual annotations performs better when the automatic segmentation predictions rather than the raters' annotations were tested. For this study, we used the ISBI 2015 Multiple Sclerosis (MS) challenge dataset, which includes annotations by two raters with different levels of expertise. The results obtained allow us to underline a worrisome clinical implication of a DNN bias induced by an inter-rater bias during training. Specially, we show that the differences in MS-lesion load estimates increase when the volume calculations are done based on the DNNs' segmentation predictions instead of the manual annotations used for training.

Motivation & Objective

• To investigate the impact of inter-rater bias—distinct from random variability—on deep learning-based medical image segmentation.
• To assess whether differences in rater annotations during training and inference affect DNN performance, particularly in terms of Dice score and lesion load estimation.
• To evaluate whether DNNs learn and amplify rater-specific biases present in training data.
• To explore the clinical implications of such biases, especially in quantitative MRI analysis of multiple sclerosis lesions.

Proposed method

• Utilized the ISBI 2015 Multiple Sclerosis challenge dataset, containing two rater annotations with varying expertise levels.
• Trained a semantic segmentation DNN on annotations from one rater and evaluated it on test data from the other rater to isolate the effect of rater bias.
• Measured performance using the Dice coefficient to quantify segmentation overlap between predictions and ground truth from different raters.
• Trained a separate classifier DNN to distinguish between raters based on their manual annotations and tested its performance on DNN predictions to assess bias propagation.
• Compared lesion load estimates derived from DNN predictions versus manual annotations to evaluate bias amplification in quantitative metrics.

Experimental results

Research questions

• RQ1 How does inter-rater bias in manual annotations affect the performance of DNNs in semantic segmentation of medical images?
• RQ2 Does the Dice score decrease when test annotations come from a different rater than the training annotations?
• RQ3 To what extent do DNNs learn and propagate rater-specific biases from training data to predictions?
• RQ4 Can a classifier trained to distinguish raters perform better on DNN predictions than on original manual annotations, indicating bias amplification?
• RQ5 How does inter-rater bias affect quantitative lesion load estimation when using DNN predictions instead of manual annotations?

Key findings

• Lower Dice scores were observed when test annotations came from a different rater than the training annotations, indicating performance degradation due to inter-rater bias.
• The DNN classifier achieved higher accuracy in distinguishing raters when using DNN predictions rather than manual annotations, suggesting that the model amplifies rater-specific biases.
• Inter-rater bias in training data was amplified in the DNN's segmentation predictions, leading to increased differences in lesion load estimates between raters.
• Quantitative lesion load estimates derived from DNN predictions showed greater variability across raters than those from manual annotations, indicating a clinically concerning bias amplification.
• The results demonstrate that DNNs are not immune to rater bias and can propagate and magnify differences between raters, even when trained on deterministic models.