[论文解读] Improving the repeatability of deep learning models with Monte Carlo dropout

Andréanne Lemay, Katharina Hoebel+8 more|arXiv (Cornell University)|Feb 15, 2022

Artificial Intelligence in Healthcare and Education被引用 5

一句话总结

本文提出在测试阶段使用蒙特卡洛（MC）丢弃法，以提升深度学习模型在医学图像分类中的可重复性。通过在相同输入图像上进行多次前向传播并激活丢弃层，该方法显著降低了对同一患者重复图像的预测变异性，在95%一致性范围内的可重复性提升16个百分点，并改善了校准效果——尤其在二分类、多分类和有序分类任务中表现优异，且准确率损失极小。

ABSTRACT

The integration of artificial intelligence into clinical workflows requires reliable and robust models. Repeatability is a key attribute of model robustness. Repeatable models output predictions with low variation during independent tests carried out under similar conditions. During model development and evaluation, much attention is given to classification performance while model repeatability is rarely assessed, leading to the development of models that are unusable in clinical practice. In this work, we evaluate the repeatability of four model types (binary classification, multi-class classification, ordinal classification, and regression) on images that were acquired from the same patient during the same visit. We study the performance of binary, multi-class, ordinal, and regression models on four medical image classification tasks from public and private datasets: knee osteoarthritis, cervical cancer screening, breast density estimation, and retinopathy of prematurity. Repeatability is measured and compared on ResNet and DenseNet architectures. Moreover, we assess the impact of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability. Leveraging Monte Carlo predictions significantly increased repeatability for all tasks on the binary, multi-class, and ordinal models leading to an average reduction of the 95\% limits of agreement by 16% points and of the disagreement rate by 7% points. The classification accuracy improved in most settings along with the repeatability. Our results suggest that beyond about 20 Monte Carlo iterations, there is no further gain in repeatability. In addition to the higher test-retest agreement, Monte Carlo predictions were better calibrated which leads to output probabilities reflecting more accurately the true likelihood of being correctly classified.

研究动机与目标

• 解决临床深度学习中对模型可重复性关注不足的严重问题，尽管其对可靠决策支持至关重要。
• 评估测试阶段的蒙特卡洛丢弃法是否能提升多种医学图像任务及模型类型下的可重复性。
• 比较MC丢弃法在ResNet与DenseNet架构上对分类性能、可重复性及校准的影响。
• 确定最大化可重复性的MC采样迭代次数，避免收益递减。

提出的方法

• 在推理阶段应用蒙特卡洛丢弃法，通过在相同输入图像上执行多次前向传播并激活丢弃层。
• 利用MC预测结果的均值与方差估计不确定性，提升预测稳定性。
• 通过Bland-Altman图评估可重复性，使用95%一致性范围（LoA）和同一患者成对图像的分类不一致率作为指标。
• 使用Brier评分评估模型校准性，判断预测概率是否真实反映分类正确性的可能性。
• 在四个医学影像任务（膝骨关节炎、宫颈癌筛查、乳腺密度、早产儿视网膜病变）上评估四种模型类型（二分类、多分类、有序分类与回归）。
• 采用非参数一致性范围与自 resampling t 检验评估统计显著性，模型性能在1至50次MC迭代间进行比较。

实验结果

研究问题

• RQ1 测试阶段的蒙特卡洛丢弃法是否能提升多样化医学图像分类任务中深度学习模型的可重复性？
• RQ2 MC丢弃法在提升可重复性的同时，对分类准确率与校准性有何影响？
• RQ3 为实现可重复性最大化，MC采样迭代的最优次数是多少，且不会出现收益递减？
• RQ4 不同模型架构（ResNet、DenseNet）与模型类型（二分类、多分类、有序分类、回归）对MC丢弃法的响应如何？
• RQ5 在相同患者、相似条件下重复测试图像时，MC丢弃法在多大程度上降低了预测变异性？

主要发现

• 蒙特卡洛丢弃法在所有任务中均显著提升了可重复性，95%一致性范围平均缩小16个百分点。
• 使用MC丢弃法后，分类不一致率下降7个百分点，表明重复图像上的预测更加一致。
• 超过20次MC迭代后，可重复性不再提升，表明该点后出现收益递减。
• MC丢弃法改善了模型校准性，使预测概率更贴近真实分类正确性的可能性。
• 二分类、多分类与有序分类模型在可重复性与准确率方面均表现出一致改进；回归模型则未见一致提升。
• 该方法在不同架构（ResNet18、ResNet50、DenseNet121）上均有效，且适用于所有模型类型，易于实现。