[Paper Review] Inducing Causal Structure for Interpretable Neural Networks

Atticus Geiger, Zhengxuan Wu+6 more|arXiv (Cornell University)|Dec 1, 2021

Explainable Artificial Intelligence (XAI)26 references4 citations

TL;DR

This paper introduces Interchange Intervention Training (IIT), a differentiable method that aligns neural network representations with a symbolic causal model to enforce counterfactual behavior during training. IIT ensures that neural models not only perform well on downstream tasks but also realize the intended causal structure, outperforming multi-task learning and data augmentation in both accuracy and interpretability across vision, language, and reasoning tasks.

ABSTRACT

In many areas, we have well-founded insights about causal structure that would be useful to bring into our trained models while still allowing them to learn in a data-driven fashion. To achieve this, we present the new method of interchange intervention training (IIT). In IIT, we (1) align variables in a causal model (e.g., a deterministic program or Bayesian network) with representations in a neural model and (2) train the neural model to match the counterfactual behavior of the causal model on a base input when aligned representations in both models are set to be the value they would be for a source input. IIT is fully differentiable, flexibly combines with other objectives, and guarantees that the target causal model is a causal abstraction of the neural model when its loss is zero. We evaluate IIT on a structural vision task (MNIST-PVR), a navigational language task (ReaSCAN), and a natural language inference task (MQNLI). We compare IIT against multi-task training objectives and data augmentation. In all our experiments, IIT achieves the best results and produces neural models that are more interpretable in the sense that they more successfully realize the target causal model.

Motivation & Objective

• To integrate well-founded causal knowledge into neural networks while preserving data-driven learning.
• To address the limitation of existing interpretability methods that only analyze trained models rather than actively shaping their structure.
• To develop a training objective that guarantees the neural model realizes the target causal model when optimization converges.
• To evaluate whether enforcing causal structure improves systematic generalization and model interpretability.
• To compare IIT against multi-task learning and data augmentation in terms of behavioral performance and causal fidelity.

Proposed method

• IIT aligns variables in a symbolic causal model C with representations in a neural network N.
• It trains N to match the counterfactual behavior of C by performing 'interchange interventions'—swapping internal states between inputs—using C’s output as the gold label.
• The training objective minimizes the cross-entropy between the neural model’s intervention output and the causal model’s counterfactual output across input pairs.
• The method is fully differentiable and can be combined with other objectives, such as multi-task learning or data augmentation.
• Causal structure is enforced via intervention-based supervision, where the model is trained to respond consistently to counterfactual perturbations in aligned representations.
• The approach guarantees that, when the loss is zero, the causal model is a causal abstraction of the neural network (Beckers & Halpern, 2019).

Experimental results

Research questions

• RQ1 Can we train neural networks to realize a predefined symbolic causal structure while maintaining high performance?
• RQ2 Does enforcing causal structure via IIT lead to better systematic generalization than standard training or data augmentation?
• RQ3 How does IIT compare to multi-task learning and data augmentation in terms of both behavioral accuracy and interpretability?
• RQ4 Can IIT be flexibly combined with other training objectives to further improve model performance and structure?
• RQ5 To what extent does IIT ensure that the neural model’s internal representations reflect the intended causal abstraction?

Key findings

• On the MQNLI task, IIT achieved near-perfect behavioral accuracy (≈100%) and interchange intervention accuracy (≈100%), indicating strong alignment with the target causal model.
• Data augmentation achieved near-perfect behavioral accuracy but only 30–40% interchange intervention accuracy, showing it fails to produce interpretable models.
• Multi-task learning combined with IIT further improved performance, suggesting synergistic benefits.
• Aligning QPObj to BERT layers 6 and later yielded the best results, with both high behavioral and intervention accuracy.
• On ReaSCAN and MNIST-PVR, IIT outperformed multi-task and data augmentation baselines in systematic generalization and causal fidelity.
• IIT successfully induced the intended causal structure across diverse tasks, demonstrating its generalizability and effectiveness in creating interpretable neural networks.