[Paper Review] On Completeness-aware Concept-Based Explanations in Deep Neural Networks

Chih‐Kuan Yeh, Been Kim+4 more|arXiv (Cornell University)|Oct 17, 2019

Explainable Artificial Intelligence (XAI)Computer Science37 references50 citations

TL;DR

The paper defines a completeness score for concept-based explanations in DNNs, introduces a completeness-aware concept discovery method with an interpretability regularizer, and proposes ConceptSHAP to quantify concept attributions; it validates the approach on synthetic, image (AwA), and text (IMDB) data.

ABSTRACT

Human explanations of high-level decisions are often expressed in terms of key concepts the decisions are based on. In this paper, we study such concept-based explainability for Deep Neural Networks (DNNs). First, we define the notion of completeness, which quantifies how sufficient a particular set of concepts is in explaining a model's prediction behavior based on the assumption that complete concept scores are sufficient statistics of the model prediction. Next, we propose a concept discovery method that aims to infer a complete set of concepts that are additionally encouraged to be interpretable, which addresses the limitations of existing methods on concept explanations. To define an importance score for each discovered concept, we adapt game-theoretic notions to aggregate over sets and propose ConceptSHAP. Via proposed metrics and user studies, on a synthetic dataset with apriori-known concept explanations, as well as on real-world image and language datasets, we validate the effectiveness of our method in finding concepts that are both complete in explaining the decisions and interpretable. (The code is released at https://github.com/chihkuanyeh/concept_exp)

Motivation & Objective

• Define a formal completeness score for concept-based explanations in DNNs.
• Develop a complete, interpretable set of concepts via unsupervised discovery.
• Propose ConceptSHAP to quantify concept attribution under completeness.
• Regularize discovery to ensure coherence and semantic meaningfulness of concepts.
• Demonstrate effectiveness on synthetic data and real-world image and language datasets.

Proposed method

• Represent input x as patches x_t and project onto concept vectors c_1,...,c_m in a latent space.
• Define concept product v_c(x_t) via thresholded inner products with c_j and normalize to form v_c(x).
• Assume complete concepts yield a sufficient statistic for predictions; learn a mapping g from v_c(x) to the activation space and evaluate predictiveness.
• Propose a regularizer R(c) that promotes locality/consistency of concept neighbors and inter-concept diversity to improve interpretability.
• Optimize a joint objective log P[h_y(g(v_c(x)))] + R(c) with SGD to discover concepts and the mapping g.
• Define ConceptSHAP as a Shapley-value-based attribution of each concept to the completeness score, including per-class variants for multi-class settings.

Experimental results

Research questions

• RQ1 How can we quantify the sufficiency (completeness) of a set of concepts in explaining a DNN decision?
• RQ2 Can we automatically discover a complete and interpretable set of concepts that jointly explain model predictions?
• RQ3 How to attribute the importance of each concept toward the overall completeness score (and per class) in a principled way?

Key findings

• The proposed completeness score eta_f(c_1,...,c_m) measures how well concept scores recover model predictions relative to the full model.
• The completeness-aware discovery method outperforms baselines (ACE, ACE-SP, PCA, k-means) in retrieving correct concepts and achieving higher automated alignment in a synthetic dataset.
• On Animals with Attributes (AwA) and synthetic data, the method achieves the highest completeness across methods.
• ConceptSHAP provides per-concept attribution to the completeness score, upholding Shapley axioms (efficiency, symmetry, dummy, additivity).
• A per-class ConceptSHAP variant identifies concepts most contributing to specific classes, aiding class-specific interpretability.
• Human and automated evaluations show that discovered concepts are coherent, interpretable, and semantically meaningful across image and language tasks (AwA and IMDB).