[Paper Review] Towards Automatic Concept-based Explanations

Amirata Ghorbani, James Wexler+2 more|arXiv (Cornell University)|Feb 7, 2019

Explainable Artificial Intelligence (XAI)Computer Science47 references86 citations

TL;DR

ACE automatically discovers human-understandable concepts from data by clustering segments across multi-resolution images and measuring their importance with TCAV, enabling global, concept-based explanations of neural networks.

ABSTRACT

Interpretability has become an important topic of research as more machine learning (ML) models are deployed and widely used to make important decisions. Most of the current explanation methods provide explanations through feature importance scores, which identify features that are important for each individual input. However, how to systematically summarize and interpret such per sample feature importance scores itself is challenging. In this work, we propose principles and desiderata for \emph{concept} based explanation, which goes beyond per-sample features to identify higher-level human-understandable concepts that apply across the entire dataset. We develop a new algorithm, ACE, to automatically extract visual concepts. Our systematic experiments demonstrate that \alg discovers concepts that are human-meaningful, coherent and important for the neural network's predictions.

Motivation & Objective

• Define desiderata for concept-based explanations that are meaningful, coherent, and locally important for predictions.
• Automatically discover high-level concepts from data without human labeling of concepts.
• Demonstrate that discovered concepts are human-meaningful, coherent, and predictive for the model.
• Provide a scalable framework applicable to a widely-used CNN on ImageNet.
• Offer human-in-the-loop validation via experiments to validate concepts and their importance.

Proposed method

• Input a trained classifier and a set of class-specific images to ACE.
• Segment images at multiple resolutions to capture simple to complex concepts.
• Map segments into a CNN activation space and cluster them to form concepts, removing outliers.
• Use TCAV to compute a concept’s importance for the target class.
• Evaluate concept meaningfulness and coherency with human experiments and analyze concept-based importance via SSC/SDC ideas.

Experimental results

Research questions

• RQ1 Can ACE automatically discover meaningful, coherent concepts from data without manual concept labeling?
• RQ2 Do the extracted concepts meaningfully reflect what the model uses for its predictions?
• RQ3 How do concept-level importance scores (TCAV) behave when concepts are added or removed?
• RQ4 What qualitative and quantitative insights about the model emerge from ACE-discovered concepts?

Key findings

• ACE discovers concepts at multiple abstraction levels (textures, parts, objects) that are coherent and often interpretable to humans.
• Human experiments show discovered concepts are coherent and meaningful across individuals.
• Top concepts significantly influence predictions; removing the top concepts substantially degrades accuracy on evaluated images.
• For 100 ImageNet classes, top-5 ACE concepts suffice to reach about 80% of original accuracy when selectively added or removed.
• Stitching important concepts on blank images can still lead to correct class predictions, indicating concepts capture salient model cues.