[Paper Review] Clinical acceptance of software based on artificial intelligence technologies (radiology)

С. П. Морозов, Anton V. Vladzymyrskyy+5 more|arXiv (Cornell University)|Aug 1, 2019

Artificial Intelligence in Healthcare and Education1 references12 citations

TL;DR

This paper proposes a methodological framework for clinical testing, acceptance, and scientific evaluation of AI-based software in radiology, focusing on accuracy and efficiency assessment to support regulatory registration as a medical device. It outlines standardized procedures for validating AI algorithms through structured clinical trials and performance benchmarks.

ABSTRACT

Aim: provide a methodological framework for the process of clinical tests, clinical acceptance, and scientific assessment of algorithms and software based on the artificial intelligence (AI) technologies. Clinical tests are considered as a preparation stage for the software registration as a medical product. The authors propose approaches to evaluate accuracy and efficiency of the AI algorithms for radiology.

Motivation & Objective

• Address the lack of standardized procedures for evaluating AI-based software in radiology prior to clinical deployment.
• Develop a structured approach to clinical testing that prepares AI software for regulatory registration as a medical device.
• Ensure scientific rigor and clinical relevance in assessing the accuracy and efficiency of AI algorithms in medical imaging.
• Provide a reproducible and transparent evaluation pipeline for AI systems in radiology to support regulatory and clinical adoption.
• Establish criteria for performance benchmarking and validation that align with medical device standards and clinical needs.

Proposed method

• Propose a multi-stage clinical testing framework for AI software, beginning with pre-clinical validation and progressing to controlled clinical trials.
• Integrate performance metrics such as sensitivity, specificity, and area under the ROC curve (AUC) to evaluate diagnostic accuracy.
• Define standardized data collection and annotation protocols to ensure consistency and reproducibility across studies.
• Implement a tiered evaluation process involving independent validation on diverse, real-world imaging datasets.
• Apply statistical methods to assess algorithm reliability, generalizability, and robustness across different clinical settings.
• Establish a feedback loop between clinical performance and algorithm refinement to support iterative improvement.

Experimental results

Research questions

• RQ1 What methodological framework ensures reliable and reproducible clinical evaluation of AI-based radiology software?
• RQ2 How can AI algorithms be systematically tested for accuracy and efficiency before regulatory approval?
• RQ3 What criteria define clinical acceptability of AI software in radiology from a regulatory and medical practice perspective?
• RQ4 How can performance metrics be standardized across different AI systems and imaging modalities?
• RQ5 What role does clinical validation play in transitioning AI software from research to regulated medical use?

Key findings

• The proposed framework enables systematic clinical testing of AI software, positioning it as a prerequisite for regulatory registration as a medical device.
• Standardized performance metrics such as AUC, sensitivity, and specificity are essential for objective evaluation of diagnostic accuracy.
• Clinical validation on diverse, real-world datasets improves the generalizability and reliability of AI algorithms in radiology.
• The framework supports iterative refinement of AI models through feedback from clinical performance data.
• Clear separation between pre-clinical validation and clinical trials enhances transparency and scientific credibility.
• The methodological approach aligns with international standards for medical device evaluation, facilitating regulatory acceptance.