[Paper Review] Explainable Deep Learning: A Field Guide for the Uninitiated

Gabriëlle Ras, Ning Xie+2 more|arXiv (Cornell University)|Apr 30, 2020

Explainable Artificial Intelligence (XAI)92 citations

TL;DR

This paper presents a field guide introducing a simple three-dimensional taxonomy of explainable DNN methods, evaluation approaches, and practical design considerations for newcomers to the field.

ABSTRACT

Deep neural networks (DNNs) have become a proven and indispensable machine learning tool. As a black-box model, it remains difficult to diagnose what aspects of the model's input drive the decisions of a DNN. In countless real-world domains, from legislation and law enforcement to healthcare, such diagnosis is essential to ensure that DNN decisions are driven by aspects appropriate in the context of its use. The development of methods and studies enabling the explanation of a DNN's decisions has thus blossomed into an active, broad area of research. A practitioner wanting to study explainable deep learning may be intimidated by the plethora of orthogonal directions the field has taken. This complexity is further exacerbated by competing definitions of what it means ``to explain'' the actions of a DNN and to evaluate an approach's ``ability to explain''. This article offers a field guide to explore the space of explainable deep learning aimed at those uninitiated in the field. The field guide: i) Introduces three simple dimensions defining the space of foundational methods that contribute to explainable deep learning, ii) discusses the evaluations for model explanations, iii) places explainability in the context of other related deep learning research areas, and iv) finally elaborates on user-oriented explanation designing and potential future directions on explainable deep learning. We hope the guide is used as an easy-to-digest starting point for those just embarking on research in this field.

Motivation & Objective

• Define a simple, three-dimensional space to categorize foundational explainable DNN methods.
• Summarize evaluation approaches for model explanations.
• Contextualize explainability with related deep learning research areas.
• Provide designer-oriented guidance for building explainable DNN systems.
• Highlight future directions and limitations to guide new research undertakings.

Proposed method

• Introduce a three-dimensional taxonomy of explainable DNN methods: Visualization, Model Distillation, and Intrinsic methods.
• Describe visualization techniques, including backpropagation-based and perturbation-based approaches, and common forms like saliency maps and heatmaps.
• Explain model distillation as creating a white-box model to mimic DNN behavior for interpretability.
• Describe intrinsic methods where explanations are built into the model design to co-optimize performance and explanations.
• Survey representative methods (e.g., CAM/Grad-CAM, LRP, DeepLIFT, Integrated Gradients) and their foundational ideas.
• Discuss evaluation considerations and user-oriented design implications for explainable systems.

Experimental results

Research questions

• RQ1 What is a minimal, intuitive taxonomy that classifies foundational explainable DNN methods?
• RQ2 How should explanations be evaluated and validated for trustworthiness and usefulness?
• RQ3 How does explainability relate to adjacent research areas in deep learning and AI?
• RQ4 What practical considerations should designers account for when building explainable DNN systems?
• RQ5 What are the limitations and promising directions for future explainability research?

Key findings

• The paper provides a simple three-dimensional space to categorize foundational explainable DNN methods: Visualization, Model Distillation, and Intrinsic methods.
• Visualization methods are subdivided into backpropagation-based and perturbation-based approaches and are commonly presented as saliency or heatmaps.
• Foundational visualization techniques include activation maximization, deconvolution, CAM/Grad-CAM, and various relevance-based methods like LRP, DeepLIFT, and Integrated Gradients.
• Model distillation introduces a white-box surrogate model to reveal decision rules learned by the DNN.
• Intrinsic methods embed explanation into the model design, enabling joint optimization of performance and interpretability.
• The guide discusses evaluation, complementarity with related fields, and practical design considerations for end users.