[Paper Review] Guarding Against Adversarial Domain Shifts with Counterfactual Regularization.

Christina Heinze‐Deml, Nicolai Meinshausen|arXiv (Cornell University)|Oct 31, 2017

Adversarial Robustness in Machine Learning57 references32 citations

TL;DR

This paper proposes counterfactual regularization to defend against adversarial domain shifts caused by mutable style features (e.g., rotation, posture, image quality) in image classification. By modeling groups of images from the same underlying object as counterfactuals under interventions on style features, the method enforces invariance through grouping-aware regularization, improving robustness without relying on mutable features.

ABSTRACT

When training a deep network for image classification, one can broadly distinguish between two types of latent features of images that will drive the classification: (i) immutable or core features that are inherent to the object in question and do not change substantially from one instance of the object to another and (ii) or features such as position, rotation or image quality but also more complex ones like hair color or posture for images of persons. The distribution of the style features can change in the future. While transfer learning would try to adapt to a shift in the distribution(s), we here want to protect against future adversarial domain shifts, arising through changing style features, by ideally not using the mutable style features altogether. There are two broad scenarios and we show how exploiting grouping information in the data helps in both. (a) If the style features are known explicitly (e.g. rotation) one usually proceeds by using data augmentation. By exploiting the grouping information about which original image an augmented sample belongs to, we can reduce the sample size required to achieve invariance to the style feature in question. (b) Sometimes the style features are not known explicitly but we still have information about samples that belong to the same underlying object (such as different pictures of the same person). By constraining the classification to give the same forecast for all instances that belong to the same object, we show how using this grouping information leads to invariance to such implicit style features and helps to protect against adversarial domain shifts. We provide a causal framework for the problem and treat groups of instances of the same object as counterfactuals under different interventions on the mutable style features. We show links to questions of fairness, transfer learning and adversarial examples.

Motivation & Objective

• To address the challenge of adversarial domain shifts caused by changes in mutable style features such as rotation, lighting, or posture in image classification.
• To develop a method that reduces reliance on style features by enforcing invariance across variations of the same underlying object.
• To provide a causal framework for modeling style variations as interventions on counterfactual instances of the same object.
• To improve robustness in transfer learning and fairness by minimizing sensitivity to non-core, mutable image features.
• To unify concepts from fairness, adversarial robustness, and domain shift under a counterfactual regularization approach.

Proposed method

• Treat groups of images belonging to the same object as counterfactual instances under different interventions on style features (e.g., rotation, lighting).
• Use grouping information—linking augmented or variant images to their original source—to enforce consistent predictions across all variants of the same object.
• Apply a regularization loss that penalizes prediction variance across instances in the same group, promoting invariance to mutable style features.
• Formulate the problem within a structural causal model where style features are interventions on the same underlying object.
• Leverage data augmentation and implicit grouping (e.g., multiple images of the same person) to identify counterfactual samples without explicit style labels.
• Integrate counterfactual regularization into standard deep learning training to jointly optimize for accuracy and invariance.

Experimental results

Research questions

• RQ1 How can we protect deep neural networks from adversarial domain shifts caused by changes in mutable style features like rotation or image quality?
• RQ2 In the absence of explicit style annotations, how can grouping information about images of the same object be used to enforce invariance?
• RQ3 What is the role of counterfactual reasoning in modeling domain shifts due to style variations?
• RQ4 How does counterfactual regularization improve robustness in transfer learning and fairness settings?
• RQ5 Can invariance to style features be achieved without relying on explicit data augmentation or style disentanglement?

Key findings

• Counterfactual regularization significantly reduces model dependence on mutable style features by enforcing consistent predictions across group members.
• The method achieves invariance to style shifts even when style features are not explicitly known, using only grouping information.
• By modeling object groups as counterfactuals, the approach provides a causal framework that links domain shift robustness to fairness and adversarial robustness.
• The use of grouping information reduces the required sample size for achieving invariance under data augmentation.
• The method generalizes across scenarios: both known style features (via augmentation) and unknown style features (via implicit grouping) benefit from the same regularization mechanism.
• Empirical results show improved robustness to distribution shifts without compromising accuracy on the original distribution.