[Paper Review] Explaining Image Classifiers Using Contrastive Counterfactuals in Generative Latent Spaces

Kamran Alipour, Aditya Lahiri+3 more|arXiv (Cornell University)|Jun 10, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a method to generate contrastive, causal, and interpretable counterfactual explanations for black-box image classifiers using pretrained generative models in latent space, without retraining or conditioning. It enables global explanations via sufficiency and necessity scores for attributes, demonstrating effectiveness on face attractiveness classification with both feature attributions and generated counterfactual images.

ABSTRACT

Despite their high accuracies, modern complex image classifiers cannot be trusted for sensitive tasks due to their unknown decision-making process and potential biases. Counterfactual explanations are very effective in providing transparency for these black-box algorithms. Nevertheless, generating counterfactuals that can have a consistent impact on classifier outputs and yet expose interpretable feature changes is a very challenging task. We introduce a novel method to generate causal and yet interpretable counterfactual explanations for image classifiers using pretrained generative models without any re-training or conditioning. The generative models in this technique are not bound to be trained on the same data as the target classifier. We use this framework to obtain contrastive and causal sufficiency and necessity scores as global explanations for black-box classifiers. On the task of face attribute classification, we show how different attributes influence the classifier output by providing both causal and contrastive feature attributions, and the corresponding counterfactual images.

Motivation & Objective

• To address the lack of causal and interpretable explanations in black-box image classifiers, especially for sensitive applications.
• To generate counterfactual images that causally influence classifier decisions while preserving human-understandable attribute changes.
• To compute probabilistic sufficiency and necessity scores for global, contrastive explanations of classifier behavior.
• To enable explanation of any black-box image classifier using pre-trained generative models independent of the classifier's training data.
• To provide a scalable, efficient alternative to coefficient-based attributions that suffer from instability and preprocessing dependence.

Proposed method

• A shift-predictor model is trained in the latent space of a pretrained generative model to predict direction vectors for desired attribute changes.
• Counterfactual images are generated by applying learned shift directions to the latent code of an input image, preserving identity while modifying specific attributes.
• Sufficiency and necessity scores are computed probabilistically by measuring the likelihood of outcome change under attribute modifications.
• The method operates in the latent space of a generative model (e.g., StyleGAN), enabling efficient sampling without retraining the generator.
• The framework supports both positive and negative attribute perturbations (e.g., increase or decrease of attributes) to compute SUF⁺, SUF⁻, NEC⁺, NEC⁻ scores.
• Explanations are grounded in contrastive reasoning: 'What if this attribute were different?' with quantified probabilities of outcome change.

Experimental results

Research questions

• RQ1 How can we generate counterfactual images that causally influence a black-box classifier’s output while ensuring interpretability?
• RQ2 What is the role of pre-trained generative models in enabling efficient, scalable, and disentangled counterfactual generation for image classifiers?
• RQ3 How can sufficiency and necessity scores be formalized and computed in the context of image classification using latent space manipulations?
• RQ4 To what extent do these scores reveal hidden biases and feature interactions in image classifiers?
• RQ5 Can contrastive explanations based on probabilistic counterfactuals outperform traditional coefficient-based attribution methods?

Key findings

• High SUF⁺ values for Heavy Makeup, Blond Hair, and Young attributes indicate that increasing these features is most sufficient to flip an unattractive prediction to attractive.
• High SUF⁻ values for Male and Bald attributes show that decreasing these features is most sufficient to reverse an unattractive outcome to attractive.
• High NEC⁺ scores for Baldness, Moustache, and Male indicate that avoiding increases in these attributes is most necessary to maintain an attractive classification.
• High NEC⁻ scores for Young, Heavy Makeup, and Blond Hair show that avoiding decreases in these attributes is most necessary to preserve an attractive label.
• The method successfully generates realistic counterfactual images that visually validate the computed sufficiency and necessity scores.
• The approach outperforms standard coefficient-based attributions by providing causally grounded, stable, and interpretable explanations independent of preprocessing.