[论文解读] Metrics reloaded: Recommendations for image analysis validation

Lena Maier‐Hein, Annika Reinke+72 more|arXiv (Cornell University)|Jun 3, 2022

Artificial Intelligence in Healthcare and Education参考文献 69被引用 62

一句话总结

本文提出 Metrics Reloaded，这是一个基于问题指纹、Delphi 驱动过程以及在线工具的图片分析验证中的面向问题的度量选择框架。

ABSTRACT

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.

研究动机与目标

• 识别为何生物医学图像分析中的度量选择常常不能反映领域需求。
• 开发一个面向问题的框架（Metrics Reloaded），用于选择验证度量。
• 创建一个结构化的问题指纹，以在图像、对象和像素层级上引导度量选择。
• 通过生物医学用例演示框架的适用性，并提供一个用于实际使用的在线工具。

提出的方法

• 通过多阶段 Delphi 过程（2020–2022）在国际专家输入下开发 Metrics Reloaded。
• 引入问题指纹来捕捉与度量选择相关的领域、数据和输出相关属性。
• 定义四个问题类别：图像级分类、目标检测、语义分割和实例分割。
• 从共识基础的参考度量库创建度量池，并为度量选择路径提供信息。
• 为模棱两可的情况提供决策指南，并实现一个在线工具以在工作流程中帮助用户。

实验结果

研究问题

• RQ1 度量选择如何与潜在的生物医学问题及领域利益保持一致？
• RQ2 一个问题指纹应捕捉哪些属性，以实现面向问题和模态无关的度量推荐？
• RQ3 如何通过 Delphi 驱动的过程产生一个强健、基于共识的图像分析验证度量库？
• RQ4 一个实用的在线工具是否能促进跨域一致的图像分析任务的验证度量选择？
• RQ5 Metrics Reloaded 框架在不同成像模态和问题规模上具有多大程度的通用性？

主要发现

• Metrics Reloaded 识别了三类度量陷阱：不恰当的问题类别、糟糕的度量选择，以及糟糕的度量应用。
• 一种问题指纹方法通过编码领域知识实现面向问题和模态无关的度量推荐。
• 该框架在四个问题类别（ImLC、SemS、ObD、InS）中提供结构化的度量路径和决策指南，并带有 Delphi 共识支持的度量库。
• 一个在线工具实现了该框架，帮助用户选择和应用适当的度量。
• 该联盟通过多个生物学和医学用例展示了框架的广泛适用性。
• 度量库既包含常用度量，也包含较少人知的参考，例如 Net Benefit 和 Expected Cost，旨在捕捉验证中的权衡。