[Paper Review] Handling Inter-Annotator Agreement for Automated Skin Lesion Segmentation

Vinícius Ribeiro, Sandra Avila+1 more|arXiv (Cornell University)|Jun 6, 2019

Cutaneous Melanoma Detection and Management25 references17 citations

TL;DR

This paper investigates inter-annotator agreement in skin lesion segmentation using the ISIC Archive, proposing simple image-processing conditioning techniques (e.g., convex hull, morphological operations) to improve consistency in ground truth annotations. It demonstrates that such conditioning significantly enhances inter-annotator agreement—particularly increasing median Cohen’s Kappa from 0.7185 to 0.7552—without distorting lesion boundaries, offering a practical solution for training and evaluating robust segmentation models.

ABSTRACT

In this work, we explore the issue of the inter-annotator agreement for training and evaluating automated segmentation of skin lesions. We explore what different degrees of agreement represent, and how they affect different use cases for segmentation. We also evaluate how conditioning the ground truths using different (but very simple) algorithms may help to enhance agreement and may be appropriate for some use cases. The segmentation of skin lesions is a cornerstone task for automated skin lesion analysis, useful both as an end-result to locate/detect the lesions and as an ancillary task for lesion classification. Lesion segmentation, however, is a very challenging task, due not only to the challenge of image segmentation itself but also to the difficulty in obtaining properly annotated data. Detecting accurately the borders of lesions is challenging even for trained humans, since, for many lesions, those borders are fuzzy and ill-defined. Using lesions and annotations from the ISIC Archive, we estimate inter-annotator agreement for skin-lesion segmentation and propose several simple procedures that may help to improve inter-annotator agreement if used to condition the ground truths.

Motivation & Objective

• To analyze inter-annotator agreement in skin lesion segmentation, a critical but underexplored challenge in medical image segmentation.
• To evaluate how different levels of agreement affect model training and evaluation in automated skin lesion analysis.
• To propose and test simple conditioning methods that enhance agreement without distorting lesion boundaries.
• To assess whether conditioning ground truths can improve robustness and fairness in segmentation model evaluation.
• To investigate why top-performing models in recent challenges avoided using extra annotated data, potentially due to annotation inconsistency.

Proposed method

• The study uses the ISIC Archive, which provides multiple human-annotated segmentation masks per image, to estimate inter-annotator agreement using Cohen’s Kappa and Dice scores.
• It applies simple image-processing operations—bounding box, convex hull, morphological closing, and morphological opening—as conditioning techniques to modify original ground truths.
• Each conditioning method is applied to all annotations, and inter-annotator agreement is re-evaluated to assess improvement.
• Statistical hypothesis testing (Kolmogorov-Smirnov test) is used to compare agreement distributions before and after conditioning.
• The analysis focuses on the ISIC Archive and its subsets used in the 2017 and 2018 challenges to assess generalizability.
• Distributions of agreement scores are visualized using violin plots and density estimates to compare original and conditioned ground truths.

Experimental results

Research questions

• RQ1 How does inter-annotator agreement vary across different skin lesion segmentation datasets, particularly in the ISIC Archive?
• RQ2 What do varying degrees of inter-annotator agreement signify in terms of clinical and machine learning use cases?
• RQ3 Can simple image-processing conditioning techniques improve inter-annotator agreement without distorting lesion boundaries?
• RQ4 Which conditioning method yields the highest improvement in agreement, and how does this vary across different lesion types or image subsets?
• RQ5 Why might top-performing models in recent challenges avoid using extra annotated data, and could annotation inconsistency be a contributing factor?

Key findings

• The median inter-annotator agreement (Cohen’s Kappa) for the full ISIC Archive is 0.7185, rising to 0.7552 for the ISIC Challenge 2018 subset.
• Conditioning with morphological operations and convex hull significantly improves agreement, with p-values < 0.002 for the bounding box vs. convex hull comparison.
• The bounding box conditioning worsened agreement compared to other methods, indicating that strong simplification may not be beneficial.
• The Kolmogorov-Smirnov test rejected equivalence between original and all conditioned distributions (p < 10^-20), showing statistically significant shifts in agreement distributions.
• A non-negligible tail of low-agreement annotations persists even after conditioning, indicating strong divergences remain unresolved.
• The results suggest that conditioning acts as a denoising operation, preserving key lesion features while reducing annotator-specific noise, improving model robustness and evaluation fairness.