[Paper Review] What Does Explainable AI Really Mean? A New Conceptualization of Perspectives

Derek Doran, Sarah Schulz+1 more|arXiv (Cornell University)|Oct 2, 2017

Multimodal Machine Learning ApplicationsComputer Science11 references332 citations

TL;DR

The paper identifies three cross-field notions of explainable AI—opaque, interpretable, and comprehensible—and proposes a fourth, truly explainable AI that integrates automated reasoning for explanations.

ABSTRACT

We characterize three notions of explainable AI that cut across research fields: opaque systems that offer no insight into its algo- rithmic mechanisms; interpretable systems where users can mathemat- ically analyze its algorithmic mechanisms; and comprehensible systems that emit symbols enabling user-driven explanations of how a conclusion is reached. The paper is motivated by a corpus analysis of NIPS, ACL, COGSCI, and ICCV/ECCV paper titles showing differences in how work on explainable AI is positioned in various fields. We close by introducing a fourth notion: truly explainable systems, where automated reasoning is central to output crafted explanations without requiring human post processing as final step of the generative process.

Motivation & Objective

• Motivate the need for explanations in AI decisions in high-stakes contexts.
• Analyze how explainability is framed across AI subfields using corpus data.
• Define and distinguish three explainability notions applicable across disciplines.
• Propose a fourth notion—truly explainable AI—that incorporates automated reasoning for explanations.

Proposed method

• Perform corpus-based analysis of explainability terminology in ACL, NIPS, COGSCI, and ICCV/ECCV from 2007–2016.
• Use substring searches for explain, interpret, and comprehensibility to quantify explainability references.
• Illustrate contexts with word clouds to compare semantic content across fields.
• Define three notions of explainable AI: opaque, interpretable, and comprehensible, and discuss their implications.
• Argue for a fourth notion, truly explainable AI, integrating automated reasoning with symbol-based explanations.

Experimental results

Research questions

• RQ1 What are the cross-field notions of explainable AI identified in the literature?
• RQ2 How do different AI communities (vision, NLP, cognitive science) frame and use explainability terms?
• RQ3 What distinguishes opaque, interpretable, and comprehensible systems, and when is each preferable?
• RQ4 What is the role of automated reasoning in achieving truly explainable AI?

Key findings

• Explainability language and focus differ by field: COGSCI emphasizes mechanism (participants, tasks, effects), while NLP/vision focus on data, features, and algorithms.
• Interpretable models expose mathematical mappings to outputs; comprehensible models emit symbols enabling user-driven explanations; opaque models provide no insight into mechanisms.
• There is a need to move beyond post hoc explanations to a truly explainable AI that uses automated reasoning to generate explanations.
• A fourth notion—truly explainable AI—requires integrating neural-symbolic reasoning with user-facing explanations.