[论文解读] Biomedical image analysis competitions: The state of current participation practice

Matthias Eisenmann, Annika Reinke+354 more|arXiv (Cornell University)|Dec 16, 2022

Radiomics and Machine Learning in Medical ImagingMedicine被引用 17

一句话总结

本研究通过调查80项IEEE ISBI和MICCAI 2021挑战赛的参与者，调查了生物医学图像分析竞赛中的当前实践。结果显示，知识交流是参与的主要驱动力（70%），深度学习占据主导地位（94%的解决方案采用），主要挑战包括时间限制（32%的参与者报告时间不足）和数据规模（43%的参与者报告处理受限），其中基于图像块的训练和下采样是常见的解决方案。

ABSTRACT

The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

研究动机与目标

• 理解当前生物医学图像分析竞赛中参与实践的现状。
• 识别参与基准测试挑战的主要动机。
• 分析挑战开发中的方法选择、算法策略和技术瓶颈。
• 评估验证实践、数据处理和基础设施在竞赛表现中的作用。
• 评估深度学习、模型集成和后处理在提交解决方案中的普遍程度。

提出的方法

• 针对2021年IEEE ISBI和MICCAI期间举办的80项竞赛的所有参与者开展国际调查。
• 调查问卷分为五个部分：团队与任务信息、专业知识与工作环境、竞赛策略、算法特征以及附加细节。
• 收集了关于参与动机、时间投入、基础设施访问、模型架构、数据处理和验证技术的数据。
• 对80项具有独立排行榜的独立竞赛中，参与者中位数为72%的响应进行了分析。
• 使用描述性统计和横断面分析，总结模型开发、数据处理和验证实践中的趋势。
• 采用BIAS指南以确保调查设计与解释的方法严谨性。

实验结果

研究问题

• RQ1 参与生物医学图像分析竞赛的主要动机是什么？
• RQ2 参与者通常在方法开发上投入多少时间？其中有多大比例报告时间不足？
• RQ3 竞赛提交方案中最常见的深度学习架构和数据处理策略是什么？
• RQ4 标准验证实践（如k折交叉验证和模型集成）的采用程度如何？
• RQ5 参与者报告的主要技术瓶颈是什么，特别是关于数据规模和基础设施方面？

主要发现

• 70%的参与者表示，知识交流是主要参与动机，而奖金仅作为次要激励（16%）。
• 方法开发的中位时间投入为80小时，但仍有32%的参与者报告时间不足。
• 所有提交解决方案中94%基于深度学习，其中84%使用标准架构。
• 43%的受访者表示图像数据过大，无法完整处理，其中基于图像块的训练（69%）和下采样（37%）是最常见的解决方案。
• 仅37%的参与者在训练集上使用k折交叉验证，50%进行了模型集成，其中61%使用相同模型，39%使用异构模型。
• 48%的参与者在其预测结果后应用了后处理步骤，表明该做法在优化推理结果中较为普遍。