[Paper Review] Desiderata for Representation Learning: A Causal Perspective

Zhaoran Wang, Michael I. Jordan|arXiv (Cornell University)|Sep 8, 2021

Domain Adaptation and Few-Shot LearningComputer Science95 references17 citations

TL;DR

This paper proposes a causal framework to formalize key desiderata in representation learning—non-spuriousness, efficiency, and disentanglement—using counterfactual quantities and observable implications. It introduces computable metrics like probability of necessity/sufficiency (PN/PS) and the independence-of-support score (IOSS), enabling evaluation and optimization of representations from single observational datasets without ground-truth features.

ABSTRACT

Representation learning constructs low-dimensional representations to summarize essential features of high-dimensional data. This learning problem is often approached by describing various desiderata associated with learned representations; e.g., that they be non-spurious, efficient, or disentangled. It can be challenging, however, to turn these intuitive desiderata into formal criteria that can be measured and enhanced based on observed data. In this paper, we take a causal perspective on representation learning, formalizing non-spuriousness and efficiency (in supervised representation learning) and disentanglement (in unsupervised representation learning) using counterfactual quantities and observable consequences of causal assertions. This yields computable metrics that can be used to assess the degree to which representations satisfy the desiderata of interest and learn non-spurious and disentangled representations from single observational datasets.

Motivation & Objective

• To formalize intuitive desiderata in representation learning—such as non-spuriousness, efficiency, and disentanglement—using causal inference principles.
• To address the challenge of evaluating and optimizing representations without access to manually labeled features or intervention data.
• To develop computable, observable metrics grounded in causal theory that can assess the quality of learned representations.
• To enable representation learning algorithms that target specific desiderata using these metrics, without requiring intervention or strong structural assumptions.
• To demonstrate that disentanglement and informativeness are not in conflict, supporting the feasibility of learning both simultaneously.

Proposed method

• Formalize non-spuriousness and efficiency in supervised representation learning using Pearl’s probability of causation: probability of necessity (PN) and probability of sufficiency (PS).
• Derive observable implications of PN and PS from observational data, enabling estimation without interventions.
• Propose the CAUSAL-REP algorithm to learn representations that are both necessary and sufficient causes of the label, using PN and PS as optimization objectives.
• Define causal disentanglement as the absence of causal influence among representation dimensions, formalized via independent support under a positivity condition.
• Introduce the independence-of-support score (IOSS) as a computable metric to evaluate causal disentanglement using observable data.
• Develop a representation learning algorithm with an IOSS penalty to encourage disentangled representations in unsupervised settings, such as VAEs.

Experimental results

Research questions

• RQ1 How can non-spuriousness in representation learning be formally defined and measured using causal concepts?
• RQ2 What observable metrics can quantify the efficiency and non-spuriousness of representations in supervised representation learning?
• RQ3 How can disentanglement in unsupervised representation learning be formalized as a causal property?
• RQ4 Can the independence of support among representation dimensions serve as a reliable proxy for causal disentanglement?
• RQ5 Is there a trade-off between disentanglement and informativeness in representation learning, and can both be achieved simultaneously?

Key findings

• The probability of necessity (PN) and probability of sufficiency (PS) are identifiable from observational data under mild conditions, enabling direct measurement of efficiency and non-spuriousness.
• The CAUSAL-REP algorithm successfully learns representations that are both necessary and sufficient causes of the label, improving robustness to distributional shift.
• The IOSS metric effectively captures causal disentanglement and correlates with intervention robustness, increasing with stronger regularization.
• Experiments show that increasing IOSS regularization enhances disentanglement without degrading model fit or log-likelihood, indicating no significant trade-off with informativeness.
• The IOSS-based method outperforms existing unsupervised disentanglement algorithms in producing causally disentangled representations.
• Theoretical results establish the identifiability of representations with independent support, supporting the use of IOSS as a valid learning objective.