[Paper Review] Explainable AI applications in the Medical Domain: a systematic review

Nicoletta Prentzas, Antonis Kakas+1 more|arXiv (Cornell University)|Aug 10, 2023

Artificial Intelligence in Healthcare and Education19 citations

TL;DR

A systematic review of explainable AI (XAI) solutions in medical decision support, analyzing 198 recent articles to identify trends, techniques, and gaps.

ABSTRACT

Artificial Intelligence in Medicine has made significant progress with emerging applications in medical imaging, patient care, and other areas. While these applications have proven successful in retrospective studies, very few of them were applied in practice.The field of Medical AI faces various challenges, in terms of building user trust, complying with regulations, using data ethically.Explainable AI (XAI) aims to enable humans understand AI and trust its results. This paper presents a literature review on the recent developments of XAI solutions for medical decision support, based on a representative sample of 198 articles published in recent years. The systematic synthesis of the relevant articles resulted in several findings. (1) model-agnostic XAI techniques were mostly employed in these solutions, (2) deep learning models are utilized more than other types of machine learning models, (3) explainability was applied to promote trust, but very few works reported the physicians participation in the loop, (4) visual and interactive user interface is more useful in understanding the explanation and the recommendation of the system. More research is needed in collaboration between medical and AI experts, that could guide the development of suitable frameworks for the design, implementation, and evaluation of XAI solutions in medicine.

Motivation & Objective

• Motivate and map the use of XAI in medical decision support and related domains.
• Identify the prevalent XAI techniques and model types used in medical AI applications.
• Evaluate how explainability supports trust, user interaction, and clinical integration.
• Highlight gaps such as physician participation and collaboration between medical and AI experts.

Proposed method

• Perform a representative literature sampling of 198 articles from recent years.
• Synthesize findings on the prevalence of model-agnostic XAI techniques and deep learning models.
• Assess the role of explainability in trust, physician involvement, and UI design.

Experimental results

Research questions

• RQ1 What XAI techniques are most commonly used in medical decision support?
• RQ2 Which model families (e.g., deep learning) dominate XAI medical applications?
• RQ3 How is explainability used to promote trust and clinician acceptance?
• RQ4 Are physicians actively involved in the XAI development loop, and how is UI/UX supporting understanding?
• RQ5 What are the identified gaps and future collaboration needs between medical and AI experts?

Key findings

• Model-agnostic XAI techniques are frequently employed in medical XAI solutions.
• Deep learning models are used more than other ML models in these applications.
• Explainability is often used to promote trust, but physician participation in the loop is relatively limited.
• Visual and interactive user interfaces are more useful for understanding explanations and recommendations.