[Paper Review] A causal view of compositional zero-shot recognition

Yuval Atzmon, Felix Kreuk+2 more|arXiv (Cornell University)|Jun 25, 2020

Domain Adaptation and Few-Shot Learning64 references59 citations

TL;DR

The paper frames compositional zero-shot recognition as causal interventions and proposes a causal-inspired embedding model that learns disentangled attribute and object representations, improving unseen compositional generalization on AO-CLEVr and Zappos datasets.

ABSTRACT

People easily recognize new visual categories that are new combinations of known components. This compositional generalization capacity is critical for learning in real-world domains like vision and language because the long tail of new combinations dominates the distribution. Unfortunately, learning systems struggle with compositional generalization because they often build on features that are correlated with class labels even if they are not "essential" for the class. This leads to consistent misclassification of samples from a new distribution, like new combinations of known components. Here we describe an approach for compositional generalization that builds on causal ideas. First, we describe compositional zero-shot learning from a causal perspective, and propose to view zero-shot inference as finding "which intervention caused the image?". Second, we present a causal-inspired embedding model that learns disentangled representations of elementary components of visual objects from correlated (confounded) training data. We evaluate this approach on two datasets for predicting new combinations of attribute-object pairs: A well-controlled synthesized images dataset and a real-world dataset which consists of fine-grained types of shoes. We show improvements compared to strong baselines.

Motivation & Objective

• Motivate compositional generalization and identify distribution-shift and entanglement as key challenges in zero-shot recognition.
• Propose a causal generative model where labels are causes of images and interventions on (attribute, object) reveal unseen combinations.
• Develop an embedding-based architecture that learns disentangled, causally stable representations for attributes and objects.
• Introduce loss terms that enforce independence of core factors and invertibility of embeddings.
• Evaluate the approach on AO-CLEVr and Zappos, showing improved performance against strong baselines.

Proposed method

• Model images as generated by two elementary factors: Attribute a and Object o, with latent core features φa and φo in separate spaces ΦA and ΦO.
• Use a do-intervention framework to define pdo(x) = p(x|a,o) for evaluating which intervention caused the image.
• Learn mappings ha, ho, and g that encode attributes/objects into core features and map back to image space, enabling likelihood estimation p(x|a,o).
• Approximate the negative log-likelihood by a tractable expression l−L(a,o) that incorporates distances to attribute/object prototypes ha, ho and the image reconstruction error via g(ha,ho).
• Impose three losses: data likelihood loss (reconstruction and triplet terms), independence loss via HSIC to encourage φa and φo to be conditionally independent under interventions, and invertible embedding loss to preserve label information in embeddings.
• Train with five learned mappings: ha, ho, ga, and their inverses, using MLPs.

Experimental results

Research questions

• RQ1 Can a causal perspective with interventions on (attribute, object) enable robust zero-shot compositional recognition?
• RQ2 Do disentangled, core-feature representations φa and φo yield stable generalization to unseen attribute–object combinations?
• RQ3 How does enforcing independence between φa and φo affect recognition of novel compositions?
• RQ4 How does the proposed method compare to discriminative baselines and prior CZSR approaches on real and synthetic datasets?

Key findings

• The causal approach achieves improved unseen accuracy compared to baselines on AO-CLEVr; for example, unseen accuracy for a vanilla baseline (LE) is 26%, while the causal method reaches about 47%.
• Adding a discriminative variant (VisProd&CI) raises unseen accuracy from 19% to 38% compared to VisProd, with a trade-off in seen accuracy.
• On AO-CLEVr cross-validation, modeling core features yields better trade-offs between seen and unseen accuracy, improving the harmonic mean across unseen ratios.
• The evaluation uses AO-CLEVr (synthetic) and Zappos (real-world shoes) with open and closed (generalized) zero-shot settings, showing robust performance gains for the causal model.
• The approach includes a quantified independence constraint via HSIC to encourage φa and φo to be robust to interventions, contributing to disentanglement and generalization.