[Paper Review] Do Concept Bottleneck Models Learn as Intended?

Andrei Margeloiu, Matthew Ashman+4 more|arXiv (Cornell University)|May 10, 2021

Explainable Artificial Intelligence (XAI)8 references18 citations

TL;DR

This paper investigates whether concept bottleneck models (CBMs) learn as intended, finding that joint and sequential training fail to produce semantically meaningful concepts in input space. Using post hoc interpretability methods like Integrated Gradients, the authors show that concepts do not correspond to relevant input regions, undermining interpretability, predictability, and intervenability—suggesting independent training may be the only viable approach for current CBMs.

ABSTRACT

Concept bottleneck models map from raw inputs to concepts, and then from concepts to targets. Such models aim to incorporate pre-specified, high-level concepts into the learning procedure, and have been motivated to meet three desiderata: interpretability, predictability, and intervenability. However, we find that concept bottleneck models struggle to meet these goals. Using post hoc interpretability methods, we demonstrate that concepts do not correspond to anything semantically meaningful in input space, thus calling into question the usefulness of concept bottleneck models in their current form.

Motivation & Objective

• To evaluate whether concept bottleneck models (CBMs) meet the three core desiderata: interpretability, predictability, and intervenability.
• To investigate whether the pre-specified concepts in CBMs correspond to semantically meaningful features in input space.
• To assess the effectiveness of different training strategies—joint, sequential, and independent—for achieving the intended goals of CBMs.
• To examine the limitations of existing post hoc interpretability methods in analyzing concept representations within CBMs.
• To challenge the assumption that joint training of CBMs leads to conceptually meaningful and useful representations.

Proposed method

• The authors use post hoc interpretability techniques, including Integrated Gradients, Gradients, and SmoothGrad, to analyze feature importance in input space for each concept.
• They evaluate three training strategies: independent (training g and f separately using ground-truth concepts), sequential (using predicted concepts for f), and joint (training g and f jointly with a weighted loss).
• Saliency maps are generated from the concept layer back to input space to visualize which input regions are most influential for each concept.
• The performance of CBMs is compared using correlation metrics (R²) between predicted and true concept values, and between target predictions from different models.
• Experiments are conducted on two benchmark datasets: OAI for x-ray grading with 10 clinical concepts, and CUB for bird identification with 112 semantic attributes.
• The study compares model behavior across different concept representations (e.g., one-hot encoding) and evaluates intervenability by replacing predicted concepts with ground truth.

Experimental results

Research questions

• RQ1 Do concept bottleneck models trained with joint or sequential objectives produce concepts that correspond to semantically meaningful features in input space?
• RQ2 To what extent do CBMs satisfy the three desiderata: interpretability, predictability, and intervenability?
• RQ3 How do post hoc interpretability methods like Integrated Gradients perform in attributing importance to input features for CBM concepts?
• RQ4 Is the independent training strategy the only viable method for achieving all three desiderata in current CBM frameworks?
• RQ5 How does concept representation (e.g., one-hot vs. scalar) affect the success of intervenability and concept learning in CBMs?

Key findings

• The joint and sequential CBM variants fail to learn concepts that correspond to meaningful input regions, as shown by saliency maps that attend to entire objects rather than specific parts (e.g., wings or legs).
• For the 'wing pattern' concept, both joint and independent models assign importance to the entire bird, not just the wing, indicating poor spatial localization of concepts.
• The 'leg color' concept in bird images shows no attention to the leg across multiple saliency methods, confirming that concepts do not map to their intended visual regions.
• The correlation between predicted and true concept values is significantly lower in joint and sequential models (R² = 0.24 and 0.07, respectively) compared to the concept oracle (R² = 0.47), indicating poor concept prediction.
• The independent CBM achieves the highest correlation with the concept oracle (R² = 0.47), suggesting it best preserves concept integrity and may be the only viable training strategy for current CBMs.
• The study concludes that existing saliency methods may be ill-equipped to interpret concept representations in CBMs, highlighting a need for improved attribution techniques.