[论文解读] Robustness study of noisy annotation in deep learning based medical image segmentation

Shaode Yu, Erlei Zhang+7 more|arXiv (Cornell University)|Mar 10, 2020

Machine Learning and Data Classification参考文献 14被引用 8

一句话总结

本研究通过头颈部癌患者头颈部CT扫描，调查了深度学习模型在噪声标注下的鲁棒性。通过使用不同比例的噪声与干净的下颌标注进行模型训练，作者发现当噪声水平低于20%时，性能仅轻微下降，表明在临床环境中对标注错误具有显著鲁棒性。

ABSTRACT

Partly due to the use of exhaustive-annotated data, deep networks have achieved impressive performance on medical image segmentation. Medical imaging data paired with noisy annotation are, however, ubiquitous, but little is known about the effect of noisy annotation on deep learning-based medical image segmentation. We studied the effects of noisy annotation in the context of mandible segmentation from CT images. First, 202 images of Head and Neck cancer patients were collected from our clinical database, where the organs-at-risk were annotated by one of 12 planning dosimetrists. The mandibles were roughly annotated as the planning avoiding structure. Then, mandible labels were checked and corrected by a physician to get clean annotations. At last, by varying the ratios of noisy labels in the training data, deep learning-based segmentation models were trained, one for each ratio. In general, a deep network trained with noisy labels had worse segmentation results than that trained with clean labels, and fewer noisy labels led to better segmentation. When using 20% or less noisy cases for training, no significant difference was found on the prediction performance between the models trained by noisy or clean. This study suggests that deep learning-based medical image segmentation is robust to noisy annotations to some extent. It also highlights the importance of labeling quality in deep learning

研究动机与目标

• 评估噪声标注对基于深度学习的医学图像分割性能的影响。
• 量化噪声标注开始显著降低模型性能的阈值。
• 评估在不同噪声水平标注数据上训练的深度学习模型的鲁棒性。
• 为深度学习模型在临床影像真实世界标注错误中的容错能力提供实证证据。

提出的方法

• 从临床数据库中收集了202例头颈部CT图像，并由12名计划剂量师进行初始标注。
• 通过医生重新评估和修正获得真实标注（干净标注）。
• 系统性地改变训练集中噪声标签与干净标签的比例（从0%到100%的噪声）。
• 使用相同的架构和训练协议，为每个噪声比例训练一个深度学习分割模型。
• 使用标准分割指标（如Dice分数）在保留的测试集上评估模型性能。
• 比较在不同噪声水平下训练的模型的分割性能，以评估鲁棒性。

实验结果

研究问题

• RQ1 随着噪声标注比例的增加，深度学习模型的分割性能如何变化？
• RQ2 在何种噪声比例下，模型性能开始出现显著下降？
• RQ3 是否存在一个阈值，在此之下噪声标注对模型性能影响可忽略？
• RQ4 在噪声数据上训练的模型性能与在干净数据上训练的模型相比如何？

主要发现

• 使用20%或更少的噪声标签训练的模型，其性能与在干净标签上训练的模型相比无统计学显著差异。
• 仅当训练标签中噪声比例超过20%时，才观察到性能下降。
• 当噪声标签比例超过20%后，Dice分数随噪声比例增加而逐步下降。
• 本研究证实，基于深度学习的医学图像分割模型对中等水平的标注噪声表现出显著的鲁棒性。