[Paper Review] Learning to Intervene on Concept Bottlenecks

David Steinmann, Wolfgang Stammer+2 more|arXiv (Cornell University)|Aug 25, 2023

Data Stream Mining Techniques4 citations

TL;DR

This paper introduces Concept Bottleneck Memory Models (CB2M), a novel extension to concept bottleneck models (CBMs) that uses a two-fold memory to learn from and generalize human interventions across unseen data. By storing past mistakes and corrective interventions, CB2M automatically reapply corrections without further human feedback, significantly improving model robustness under distribution shifts and confounded data with fewer required user interactions.

ABSTRACT

While deep learning models often lack interpretability, concept bottleneck models (CBMs) provide inherent explanations via their concept representations. Moreover, they allow users to perform interventional interactions on these concepts by updating the concept values and thus correcting the predictive output of the model. Up to this point, these interventions were typically applied to the model just once and then discarded. To rectify this, we present concept bottleneck memory models (CB2Ms), which keep a memory of past interventions. Specifically, CB2Ms leverage a two-fold memory to generalize interventions to appropriate novel situations, enabling the model to identify errors and reapply previous interventions. This way, a CB2M learns to automatically improve model performance from a few initially obtained interventions. If no prior human interventions are available, a CB2M can detect potential mistakes of the CBM bottleneck and request targeted interventions. Our experimental evaluations on challenging scenarios like handling distribution shifts and confounded data demonstrate that CB2Ms are able to successfully generalize interventions to unseen data and can indeed identify wrongly inferred concepts. Hence, CB2Ms are a valuable tool for users to provide interactive feedback on CBMs, by guiding a user's interaction and requiring fewer interventions.

Motivation & Objective

• To address the inefficiency of one-time human interventions in concept bottleneck models (CBMs), which require repeated user feedback and lead to user fatigue.
• To enable CBMs to learn from past interventions and generalize them to novel, unseen examples without further human input.
• To detect model mistakes autonomously before any human feedback is available, allowing targeted and efficient querying of users.
• To provide a model-agnostic extension to CBMs that enhances interpretability and interactive learning through reusable intervention knowledge.

Proposed method

• CB2M extends standard CBMs with a two-fold memory: one for storing detected model mistakes (based on concept prediction errors), and another for storing previously applied human interventions.
• The model uses a k-nearest neighbors (k-NN) approach with k=1 to retrieve the closest past mistake and apply its corresponding intervention to similar new examples.
• A mistake detection mechanism identifies data points where the bottleneck network's concept predictions are likely incorrect, using a threshold-based method on concept activation scores.
• The memory is dynamically updated as new interventions are provided, enabling the model to adapt and improve over time without retraining from scratch.
• The framework is designed to be model-agnostic, allowing integration with any CBM architecture and supporting both intervention generalization and active learning.
• The system can detect systematic errors (e.g., in confounded data) and guide human feedback to the most informative samples, reducing the number of required interactions.

Figure 1: Reusing a CBM intervention can correct model mistakes for multiple examples. Top: CBMs generate a human interpretable concept representation to solve the final task. Human users can correct these concept predictions via targeted interventions (blue) influencing the final prediction. Bottom

Experimental results

Research questions

• RQ1 Can a CBM learn to generalize human interventions to unseen data without requiring additional human feedback?
• RQ2 How effectively can CB2M detect model mistakes in the absence of initial human interventions?
• RQ3 To what extent does CB2M improve model performance under distribution shifts and confounded training data?
• RQ4 Can the memory mechanism in CB2M reduce the number of required human interventions while maintaining or improving predictive accuracy?

Key findings

• CB2M successfully generalizes interventions to previously unseen examples, significantly improving model performance on distribution shift and confounded data scenarios.
• The model detects systematic errors—such as incorrect concept predictions in confounded data—before any human feedback is provided, enabling targeted querying.
• CB2M reduces the number of required human interventions by focusing feedback on the most problematic instances, improving interaction efficiency.
• The two-fold memory mechanism enables automatic correction of repeated mistakes without retraining, demonstrating sustained performance improvement from minimal feedback.
• Experiments show that CB2M outperforms standard CBMs in scenarios involving concept drift and spurious correlations, particularly when interventions are sparse.

Figure 2: Overview of CB2M to detect mistakes or generalize interventions. A vanilla CBM (grey) is extended with a two-fold memory (orange and green). The memory compares encodings of new samples to known mistakes to (i) detect model errors or (ii) automatically correct the model via reuse of interv