[Paper Review] Variability Matters : Evaluating inter-rater variability in histopathology for robust cell detection

Cholmin Kang, Chunggi Lee+3 more|arXiv (Cornell University)|Oct 11, 2022

AI in cancer detection4 citations

TL;DR

This paper proposes a method to measure and reduce inter-rater variability in histopathology cell annotation by using an anchor pathologist to assess conformity of 120 board-certified pathologists. It demonstrates that models trained on smaller, low-variability datasets outperform those trained on larger, high-variability datasets, proving that annotation quality trumps quantity in deep learning for cell detection.

ABSTRACT

Large annotated datasets have been a key component in the success of deep learning. However, annotating medical images is challenging as it requires expertise and a large budget. In particular, annotating different types of cells in histopathology suffer from high inter- and intra-rater variability due to the ambiguity of the task. Under this setting, the relation between annotators' variability and model performance has received little attention. We present a large-scale study on the variability of cell annotations among 120 board-certified pathologists and how it affects the performance of a deep learning model. We propose a method to measure such variability, and by excluding those annotators with low variability, we verify the trade-off between the amount of data and its quality. We found that naively increasing the data size at the expense of inter-rater variability does not necessarily lead to better-performing models in cell detection. Instead, decreasing the inter-rater variability with the expense of decreasing dataset size increased the model performance. Furthermore, models trained from data annotated with lower inter-labeler variability outperform those from higher inter-labeler variability. These findings suggest that the evaluation of the annotators may help tackle the fundamental budget issues in the histopathology domain

Motivation & Objective

• To investigate the impact of inter-rater variability on deep learning model performance in histopathology cell detection.
• To develop a scalable method for measuring and quantifying annotator conformity relative to a reference (anchor) pathologist.
• To evaluate whether reducing inter-rater variability through selective data curation improves model generalization and performance.
• To provide evidence that high-quality, smaller datasets can outperform large, noisy datasets in medical image analysis.
• To support more efficient budget allocation in pathology annotation by prioritizing expert consistency over sheer data volume.

Proposed method

• Anchored annotation protocol: a single expert (anchor) labels a control set of 29,387 patches from 12,326 whole slide images (WSIs).
• Each of the 120 pathologists annotated both test and control images, with control images used to compute conformity scores.
• A modified F1-score (mF1) was used to measure each annotator’s conformity to the anchor, quantifying inter-rater variability.
• Annotators were ranked by conformity, and data was segmented into percentiles (e.g., p0–25, p75–100) to compare model performance across variability levels.
• Deep learning models were trained on subsets of data grouped by conformity percentiles, and performance was evaluated using standard metrics with statistical significance testing.
• Statistical comparison (t-test, α=0.05) was used to validate differences in model performance across data subsets with varying inter-rater variability.

Experimental results

Research questions

• RQ1 Does inter-rater variability in histopathology cell annotation significantly affect deep learning model performance?
• RQ2 Can a reference (anchor) pathologist be used to objectively measure and rank annotator conformity?
• RQ3 Is there a trade-off between dataset size and annotation quality in terms of model performance?
• RQ4 Can a smaller, high-conformity dataset outperform a larger, low-conformity dataset in cell detection tasks?
• RQ5 To what extent does excluding high-variability annotators improve model generalization and efficiency?

Key findings

• The average conformity score among 120 pathologists was 0.70 with a standard deviation of 0.08, indicating substantial inter-rater variability.
• The mean Cohen’s kappa score was κ = 0.43, reflecting only moderate agreement across annotators, with a range from -0.01 to 0.87.
• Models trained on data from the lowest-variability annotators (p75–100) outperformed those trained on the highest-variability data (p0–25), with a statistically significant p-value of 0.00614.
• Even with only 50% of the data, models trained on low-variability annotations (p50–100) outperformed models trained on the full dataset, with a p-value of 0.00054.
• The model trained on the upper 75% of data (p25–100) significantly outperformed the full-dataset model (p-value = 0.01778), demonstrating that reduced data volume with lower variability yields better performance.
• The study suggests a potential 50% reduction in annotation time—from 2298 to 1176 hours—by selecting only high-conformity annotators without sacrificing model performance.