[Paper Review] Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)

Been Kim, Martin Wattenberg+5 more|arXiv (Cornell University)|Nov 30, 2017

Explainable Artificial Intelligence (XAI)732 citations

TL;DR

The paper introduces Concept Activation Vectors (CAVs) and Testing with CAVs (TCAV) to quantify how human-defined concepts influence neural network predictions, with statistical validation and multiple applications.

ABSTRACT

The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of human-friendly concepts. The key idea is to view the high-dimensional internal state of a neural net as an aid, not an obstacle. We show how to use CAVs as part of a technique, Testing with CAVs (TCAV), that uses directional derivatives to quantify the degree to which a user-defined concept is important to a classification result--for example, how sensitive a prediction of "zebra" is to the presence of stripes. Using the domain of image classification as a testing ground, we describe how CAVs may be used to explore hypotheses and generate insights for a standard image classification network as well as a medical application.

Motivation & Objective

• Define human-interpretable concepts as sets of example data that may be external to the model's training data.
• Learn Concept Activation Vectors as directions in activation space that represent these concepts.
• Quantify the influence of a concept on a class prediction via directional derivatives (TCAV scores).
• Provide statistical tests to validate that CAVs meaningfully relate to model outputs.
• Demonstrate global (class-level) interpretability and apply to real-world scenarios including a medical imaging task.

Proposed method

• Define a user-specified concept C by collecting a positive example set P_C and a negative set N of activations at a chosen layer l.
• Train a linear classifier to separate activations f_l(x) for P_C versus N; take the normal to the decision boundary as the Concept Activation Vector v_C^l.
• Compute conceptual sensitivity S_{C,k,l}(x) as the directional derivative of the class logit h_{l,k} in the direction of v_C^l, i.e., S_{C,k,l}(x) = ∇ h_{l,k}(f_l(x)) · v_C^l.
• Aggregate across all inputs X_k of class k to form TCAV score: TCAV_q_{C,k,l} = fraction of x in X_k with S_{C,k,l}(x) > 0.
• Perform statistical significance testing by repeating CAV learning across many randomizations and testing TCAV scores against 0.5 with a Bonferroni correction.
• Optionally extend to Relative CAVs to compare two concepts C and D using a vector v_{C,D} indicating a 1-D subspace.

Experimental results

Research questions

• RQ1 Can human-defined concepts learned from outside the training data be used to explain and audit NN predictions at the class level?
• RQ2 Do CAVs provide stable, statistically significant associations with model outputs across multiple runs?
• RQ3 Where in the network are concepts learned, and how do concept influences vary across layers?
• RQ4 How do TCAV-based explanations compare to saliency-based explanations in terms of human interpretability and fidelity?
• RQ5 Can TCAV be applied to real-world tasks (e.g., medical imaging) to uncover and potentially rectify model biases or errors?

Key findings

• CAVs align with intended concepts and can reveal biases in popular networks without retraining.
• TCAV scores higher near the output layer, suggesting more direct influence of concepts on predictions in later layers.
• Statistical testing reduces spurious concept associations; many CAVs pass significance tests while some do not, filtering out irrelevant concepts.
• Compared to saliency maps, TCAV better communicated concept relevance in human experiments and aligned with ground-truth concept usage in controlled setups.
• Applied to diabetic retinopathy, TCAV highlighted concepts relevant to different DR levels and aided interpretation of model errors.
• Relative CAVs enable fine-grained comparisons between closely related concepts, aiding nuanced interpretation.