[Paper Review] Metrics reloaded: Recommendations for image analysis validation

Lena Maier‐Hein, Annika Reinke+72 more|arXiv (Cornell University)|Jun 3, 2022

Artificial Intelligence in Healthcare and EducationMedicine69 references62 citations

TL;DR

The paper introduces Metrics Reloaded, a framework for problem-aware metric selection in image analysis validation, based on problem fingerprinting, a Delphi-driven process, and an online tool.

ABSTRACT

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. Particularly in automatic biomedical image analysis, chosen performance metrics often do not reflect the domain interest, thus failing to adequately measure scientific progress and hindering translation of ML techniques into practice. To overcome this, our large international expert consortium created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. The framework was developed in a multi-stage Delphi process and is based on the novel concept of a problem fingerprint - a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), data set and algorithm output. Based on the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as a classification task at image, object or pixel level, namely image-level classification, object detection, semantic segmentation, and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool, which also provides a point of access to explore weaknesses, strengths and specific recommendations for the most common validation metrics. The broad applicability of our framework across domains is demonstrated by an instantiation for various biological and medical image analysis use cases.

Motivation & Objective

• Identify why metric choices in biomedical image analysis often fail to reflect domain needs.
• Develop a problem-aware framework (Metrics Reloaded) for selecting validation metrics.
• Create a structured problem fingerprint to guide metric choice across image, object, and pixel levels.
• Demonstrate framework applicability through biomedical use cases and provide an online tool for practical use.

Proposed method

• Developed Metrics Reloaded via a multi-stage Delphi process (2020–2022) with international expert input.
• Introduce problem fingerprinting to capture domain-, data-, and output-related properties relevant to metric selection.
• Define four problem categories: image-level classification, object detection, semantic segmentation, and instance segmentation.
• Create metric pools from a consensus-based repository of reference-based metrics and inform metric selection paths.
• Provide decision guides for ambiguous cases and implement an online tool to assist users through the workflow.

Experimental results

Research questions

• RQ1 How can metric choices be aligned with the underlying biomedical problem and domain interests?
• RQ2 What properties should a problem fingerprint capture to enable problem- and modality-agnostic metric recommendations?
• RQ3 How can a Delphi-driven process yield a robust, consensus-based pool of validation metrics for image analysis?
• RQ4 Can a practical online tool facilitate consistent, cross-domain validation metric selection across image analysis tasks?
• RQ5 To what extent is the Metrics Reloaded framework generalizable to different imaging modalities and problem scales?

Key findings

• Metrics Reloaded identifies three types of metric pitfalls: inappropriate problem category, poor metric selection, and poor metric application.
• A problem fingerprinting approach enables problem- and modality-agnostic metric recommendations by encoding domain knowledge.
• The framework provides structured metric-paths and decision guides across four problem categories (ImLC, SemS, ObD, InS) with a Delphi-consensus-backed metric pool.
• An online tool implements the framework to assist users in selecting and applying appropriate metrics.
• The consortium demonstrates broad applicability of the framework through multiple biological and medical use cases.
• The metric pool includes both common metrics and lesser-known references like Net Benefit and Expected Cost, designed to capture tradeoffs in validation.