[Paper Review] Objective task-based evaluation of artificial intelligence-based medical imaging methods: Framework, strategies and role of the physician

Abhinav K. Jha, Kyle J. Myers+6 more|arXiv (Cornell University)|Jul 9, 2021

Radiomics and Machine Learning in Medical Imaging61 references7 citations

TL;DR

This paper proposes a standardized, objective, task-based evaluation framework for AI-based medical imaging methods, with a focus on PET and neural network applications. It outlines practical evaluation strategies, emphasizes the critical role of physicians in study design and interpretation, and provides a transferable methodology applicable across imaging modalities and AI techniques.

ABSTRACT

Artificial intelligence (AI)-based methods are showing promise in multiple medical-imaging applications. Thus, there is substantial interest in clinical translation of these methods, requiring in turn, that they be evaluated rigorously. In this paper, our goal is to lay out a framework for objective task-based evaluation of AI methods. We will also provide a list of tools available in the literature to conduct this evaluation. Further, we outline the important role of physicians in conducting these evaluation studies. The examples in this paper will be proposed in the context of PET with a focus on neural-network-based methods. However, the framework is also applicable to evaluate other medical-imaging modalities and other types of AI methods.

Motivation & Objective

• To establish a robust, objective framework for evaluating AI-based medical imaging methods in clinical settings.
• To address the growing need for reliable, standardized evaluation as AI methods advance toward clinical translation.
• To highlight the essential role of physicians in defining clinical tasks, selecting appropriate metrics, and ensuring relevance of evaluation outcomes.
• To provide a comprehensive list of existing tools and strategies for implementing task-based evaluation in practice.
• To extend the framework beyond PET and neural networks to other imaging modalities and AI methodologies.

Proposed method

• Proposes a task-based evaluation framework centered on clinically relevant diagnostic tasks, such as lesion detection or tumor burden estimation.
• Emphasizes the use of observer performance metrics like AUC (area under the ROC curve) and FOM (figure of merit) to quantify diagnostic performance.
• Integrates physician input at all stages—task definition, image interpretation, and metric selection—to ensure clinical relevance.
• Recommends the use of digital phantoms and realistic patient data for training and testing AI models under controlled conditions.
• Outlines strategies for comparing AI methods against radiologists and conventional methods using reader studies and statistical analysis.
• Provides a catalog of available tools and software libraries from the literature to support implementation of the evaluation framework.

Experimental results

Research questions

• RQ1 How can AI-based medical imaging methods be objectively evaluated in a way that reflects real-world clinical tasks?
• RQ2 What role should physicians play in designing and interpreting evaluation studies for AI in medical imaging?
• RQ3 What tools and methodologies are currently available to support task-based evaluation of AI in medical imaging?
• RQ4 How can the evaluation framework be generalized across different imaging modalities and AI architectures?
• RQ5 What are the key performance metrics and experimental designs that ensure reliable and reproducible evaluation outcomes?

Key findings

• The proposed framework enables objective, task-specific evaluation of AI methods, ensuring alignment with clinical decision-making needs.
• Physician involvement is critical for defining relevant diagnostic tasks and selecting appropriate performance metrics.
• Task-based evaluation using metrics like AUC and FOM provides more clinically meaningful results than traditional image quality metrics.
• The framework is adaptable to various imaging modalities, including PET, and to different AI methods such as deep learning.
• A comprehensive list of existing tools and software is provided to support implementation of the evaluation process.
• The framework supports rigorous comparison of AI methods against radiologists and conventional imaging techniques through controlled reader studies.