[Paper Review] What do we need to build explainable AI systems for the medical domain?

Andreas Holzinger, Chris Biemann+2 more|arXiv (Cornell University)|Dec 28, 2017

Explainable Artificial Intelligence (XAI)32 references632 citations

TL;DR

The paper surveys explainable-AI approaches relevant to medicine, contrasts post-hoc and ante-hoc explainability, and discusses methods and representations for images, *omics data, and text to foster transparency and trust in clinical AI systems.

ABSTRACT

Artificial intelligence (AI) generally and machine learning (ML) specifically demonstrate impressive practical success in many different application domains, e.g. in autonomous driving, speech recognition, or recommender systems. Deep learning approaches, trained on extremely large data sets or using reinforcement learning methods have even exceeded human performance in visual tasks, particularly on playing games such as Atari, or mastering the game of Go. Even in the medical domain there are remarkable results. The central problem of such models is that they are regarded as black-box models and even if we understand the underlying mathematical principles, they lack an explicit declarative knowledge representation, hence have difficulty in generating the underlying explanatory structures. This calls for systems enabling to make decisions transparent, understandable and explainable. A huge motivation for our approach are rising legal and privacy aspects. The new European General Data Protection Regulation entering into force on May 25th 2018, will make black-box approaches difficult to use in business. This does not imply a ban on automatic learning approaches or an obligation to explain everything all the time, however, there must be a possibility to make the results re-traceable on demand. In this paper we outline some of our research topics in the context of the relatively new area of explainable-AI with a focus on the application in medicine, which is a very special domain. This is due to the fact that medical professionals are working mostly with distributed heterogeneous and complex sources of data. In this paper we concentrate on three sources: images, *omics data and text. We argue that research in explainable-AI would generally help to facilitate the implementation of AI/ML in the medical domain, and specifically help to facilitate transparency and trust.

Motivation & Objective

• Motivate the need for explainable AI in medicine due to data complexity, heterogeneity, and regulatory/ethical considerations.
• Identify and discuss explainable approaches across data modalities (images, *omics data, text) to enable transparency and trust.
• Highlight the trade-offs between predictive performance and interpretability in medical AI.
• Propose methodological directions that integrate human-computer interaction with machine learning to support decision making.

Proposed method

• Differentiate post-hoc explainability (local, model-agnostic explanations like LIME) from ante-hoc explainability (interpretable-by-design models such as linear models, decision trees, GAMs).
• Describe exemplar methods: LIME, BETA, pixel-wise decomposition, and activation maximization for interpreting classifiers.
• Discuss use of generative models (RBMs, GANs) to create interpretable prototypes for class concepts.
• Explain AM-FM decompositions and their role as meaningful, visualizable features for medical images.
• Propose AM-FM ANNs and hybrid AM-FM architectures to integrate meaningful features into deep networks.
• Advocate for interpretable models (e.g., GAMs, Bayesian rule lists) and visualization strategies (deconvolutional nets, VQA-inspired explanations) to trace decisions.

Experimental results

Research questions

• RQ1 What explainable-AI approaches are suitable for medicine given data heterogeneity and clinical needs?
• RQ2 How can we design, implement, and validate explanations that are meaningful to medical professionals across images, omics, and text?
• RQ3 What is the appropriate balance between predictive performance and interpretability in medical AI systems?
• RQ4 How can human–computer interaction principles be integrated to support trust, safety, and accountability in explainable medical AI?

Key findings

• Explainability in medicine can follow post-hoc or ante-hoc paradigms, each with distinct advantages and limitations.
• AM-FM decompositions offer interpretable, visual representations that can aid understanding of medical images and support feature visualization.
• Prototype-based explanations using activation maximization can be enhanced with data-density priors or generative models to produce plausible prototypes.
• Deconvolutional networks and VQA-inspired methods can help map network activations to human-interpretable concepts and captions.
• Interpretable models like GAMs and Bayesian rule lists can achieve competitive accuracy on large medical datasets while remaining intelligible.
• Integration of AM-FM features and interpretable architectures may reduce reliance on opaque deep representations and improve transparency in medical AI.