[Paper Review] Learning Causal Semantic Representation for Out-of-Distribution Prediction

Chang Liu, Xinwei Sun+5 more|arXiv (Cornell University)|Jan 1, 2020

Domain Adaptation and Few-Shot Learning126 references48 citations

TL;DR

The paper introduces a Causal Semantic Generative model (CSG) that separates semantic factors from variation factors to achieve robust OOD prediction and domain adaptation using causal invariance, with theoretical identifiability guarantees.

ABSTRACT

Conventional supervised learning methods, especially deep ones, are found to be sensitive to out-of-distribution (OOD) examples, largely because the learned representation mixes the semantic factor with the variation factor due to their domain-specific correlation, while only the semantic factor causes the output. To address the problem, we propose a Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately, and develop methods for OOD prediction from a single training domain, which is common and challenging. The methods are based on the causal invariance principle, with a novel design in variational Bayes for both efficient learning and easy prediction. Theoretically, we prove that under certain conditions, CSG can identify the semantic factor by fitting training data, and this semantic-identification guarantees the boundedness of OOD generalization error and the success of adaptation. Empirical study shows improved OOD performance over prevailing baselines.

Motivation & Objective

• Motivate and define the problem of OOD susceptibility due to semantic-variation entanglement in representations.
• Propose a causal generative framework that separates semantic factors from variation factors for robust prediction.
• Establish theoretical identifiability of the semantic factor from a single training domain under certain conditions.
• Develop variational Bayes-based methods for OOD generalization and domain adaptation within the CSG framework.
• Demonstrate empirical improvements over baselines on real-world image classification tasks.

Proposed method

• Propose the Causal Semantic Generative Model (CSG) with p(s,v), p(x|s,v), p(y|s) to separate semantic and variation latent factors.
• Adopt the causal invariance principle: p(x|s,v) and p(y|s) are domain-invariant, while the prior p(s,v) changes across domains.
• Use a reformulated ELBO objective that relies on an auxiliary q(s,v,y|x) to enable prediction and learning without two separate inference models.
• Introduce CSG-ind and CSG-DA variants that alter the prior on (s,v) for test-domain prediction to improve OOD generalization and enable domain adaptation.
• Provide a training objective that aligns training-domain learning with test-domain prediction via a do-like intervention on the latent factors.

Experimental results

Research questions

• RQ1 Can the semantic factor s be identified from a single training domain despite correlation with the variation factor v?
• RQ2 How does semantic identification impact the bound on out-of-distribution generalization error?
• RQ3 Can the model support robust OOD generalization and domain adaptation under covariate shifts via causal invariance?
• RQ4 What are the practical advantages of using a test-domain prior (CSG-ind/CSG-DA) over inference-invariance approaches for OOD tasks?
• RQ5 How can variational Bayes be tailored to learn the CSG representation efficiently for OOD tasks?

Key findings

• CSG can identify the semantic factor under certain additive-noise and bijectivity assumptions.
• Semantic identification leads to bounded OOD generalization error and enables accurate domain-adaptation predictions.
• CSG-based methods show improved OOD performance over baselines on real-world image classification tasks.
• A causal-invariance-based training objective with test-domain priors provides reliable predictions under domain shifts.
• The approach achieves prediction using a single training domain without requiring multiple domains or interventions in training data.