[Paper Review] Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style

Julius von Kügelgen, Yash Sharma+5 more|arXiv (Cornell University)|Jun 8, 2021

Domain Adaptation and Few-Shot Learning115 references62 citations

TL;DR

The paper presents a latent-variable model separating content and style in SSL with data augmentations, proving block-identifiability of content under broad conditions and validating with causal-rich data.

ABSTRACT

Self-supervised representation learning has shown remarkable success in a number of domains. A common practice is to perform data augmentation via hand-crafted transformations intended to leave the semantics of the data invariant. We seek to understand the empirical success of this approach from a theoretical perspective. We formulate the augmentation process as a latent variable model by postulating a partition of the latent representation into a content component, which is assumed invariant to augmentation, and a style component, which is allowed to change. Unlike prior work on disentanglement and independent component analysis, we allow for both nontrivial statistical and causal dependencies in the latent space. We study the identifiability of the latent representation based on pairs of views of the observations and prove sufficient conditions that allow us to identify the invariant content partition up to an invertible mapping in both generative and discriminative settings. We find numerical simulations with dependent latent variables are consistent with our theory. Lastly, we introduce Causal3DIdent, a dataset of high-dimensional, visually complex images with rich causal dependencies, which we use to study the effect of data augmentations performed in practice.

Motivation & Objective

• Motivate understanding of why data augmentation helps SSL by framing augmentation as a latent-variable process that preserves content while changing style.
• Introduce a content–style partition of latent representations and study identifiability of the invariant content block.
• Provide theoretical identifiability results for generative and discriminative SSL under relaxed assumptions (no independence of latents).
• Develop and validate with experiments on synthetic and causal-rich image data, including the Causal3DIdent dataset.

Proposed method

• Formalize data generation and augmentation as a latent-variable model with a content block c and a style block s.
• Define content-invariance and style-change assumptions for augmentation, and model augmentations as changes in s while keeping c fixed.
• Prove block-identifiability results: Theorem 4.2 shows content identifiability in generative SSL with matching likelihoods; Theorem 4.3 shows identifiability with an invertible encoder via alignment; Theorem 4.4 shows identifiability with non-invertible encoders using max-entropy regularization.
• Link data augmentation to causal counterfactuals within a structural causal model where c influences s but not vice versa.
• Introduce and employ the Causal3DIdent dataset to study how practical augmentations align with invariant content.

Experimental results

Research questions

• RQ1 Under what conditions can SSL with data augmentations recover the invariant content partition of latent representations?
• RQ2 Can content be identified without assuming independence among latent factors, and what role do invertible versus non-invertible encoders play?
• RQ3 How do practical data augmentations relate to causal structure between content and style, and can augmentations be interpreted as counterfactuals?
• RQ4 Does max-entropy regularization enable identifiability in non-invertible encoder settings?
• RQ5 How do augmentations perform on a causally rich, high-dimensional dataset such as Causal3DIdent in isolating content?

Key findings

• SSL with data augmentations can identify the invariant content partition under specified generative and augmentation models.
• Block-identifiability holds for generative SSL (Theorem 4.2) and for discriminative SSL with invertible encoders (Theorem 4.3).
• Identifiability extends to non-invertible encoders when a max-entropy regularization term is used (Theorem 4.4).
• The theory accommodates dependent latent variables and causal influence of content on style, aligning with simulated and causal-data experiments.
• A new Causal3DIdent dataset is introduced to study identifiability under practical augmentations and causal dependencies.