[Paper Review] Generative Interventions for Causal Learning

Chengzhi Mao, Augustine Cha+4 more|arXiv (Cornell University)|Dec 22, 2020

Generative Adversarial Networks and Image Synthesis46 references4 citations

TL;DR

This paper proposes a generative intervention framework that steers pre-trained generative models (e.g., BigGAN) to perform causal interventions on visual features, removing spurious correlations from background, viewpoint, and context. By generating images with controlled nuisance variable manipulations, the method trains robust classifiers that generalize better to out-of-distribution data, achieving state-of-the-art performance on ObjectNet (39.3% top-1 accuracy) and surpassing prior methods by up to 12% on ImageNet-C.

ABSTRACT

We introduce a framework for learning robust visual representations that generalize to new viewpoints, backgrounds, and scene contexts. Discriminative models often learn naturally occurring spurious correlations, which cause them to fail on images outside of the training distribution. In this paper, we show that we can steer generative models to manufacture interventions on features caused by confounding factors. Experiments, visualizations, and theoretical results show this method learns robust representations more consistent with the underlying causal relationships. Our approach improves performance on multiple datasets demanding out-of-distribution generalization, and we demonstrate state-of-the-art performance generalizing from ImageNet to ObjectNet dataset.

Motivation & Objective

• To address the failure of discriminative models in out-of-distribution generalization due to spurious correlations from confounding factors like background and viewpoint.
• To develop a method that enables causal representation learning in natural images where randomized interventions are not feasible.
• To improve model robustness by simulating interventions on generative models to eliminate spurious correlations.
• To transfer these generative interventions to real natural images without requiring explicit latent codes.

Proposed method

• Leverages pre-trained generative models (e.g., BigGAN) to perform controlled interventions on nuisance variables such as background and viewpoint.
• Uses the generative model's disentangled latent space to steer image generation along specific intervention directions, simulating counterfactual manipulations.
• Applies neural style transfer to transfer the intervention effects (e.g., background whitening) from generated images to real natural images.
• Employs a VGG-based feature matching loss to preserve content while transferring style from intervened generated images.
• Treats the intervention as a causal operation, with theoretical justification that it tightens the causal effect bound by reducing spurious correlations.
• Trains downstream classifiers on the intervened data, encouraging attention on causal features rather than confounding context.

Experimental results

Research questions

• RQ1 Can generative models be used to simulate causal interventions on visual features in natural images where randomized experiments are infeasible?
• RQ2 How can we generate interventions that remove spurious correlations without requiring access to ground-truth causal structures?
• RQ3 To what extent do generative interventions improve out-of-distribution generalization compared to standard data augmentation?
• RQ4 Can intervention effects be transferred from generated images to real natural images without explicit latent codes?
• RQ5 Does the resulting model attend to causal features rather than spurious context, as verified through attention visualization?

Key findings

• The method achieves 39.3% top-1 accuracy on the ObjectNet dataset using ResNet152, representing a 9% improvement over the published benchmark.
• On the ImageNet-C benchmark, the method surpasses established baselines by up to 12% in performance, indicating strong robustness to distribution shift.
• Visualizations show that the model attends to the object of interest rather than spurious background context, unlike baseline models that rely on scene context.
• The generative intervention method produces tighter causal effect bounds by increasing P(x|z), as supported by theoretical analysis.
• Transferred interventions via neural style transfer successfully remove spurious correlations from natural images, even without access to their latent codes.
• The approach generalizes across categories, as interventions such as background whitening or viewpoint change are transferable across object classes.