[Paper Review] Contrastive Learning Inverts the Data Generating Process

R. Zimmermann, Yash Sharma+3 more|arXiv (Cornell University)|Feb 17, 2021

Domain Adaptation and Few-Shot Learning47 references28 citations

TL;DR

The paper proves that InfoNCE-based contrastive learning implicitly inverts the underlying data-generating process, linking contrastive learning to generative modeling and nonlinear ICA, with empirical robustness despite violated assumptions.

ABSTRACT

Contrastive learning has recently seen tremendous success in self-supervised learning. So far, however, it is largely unclear why the learned representations generalize so effectively to a large variety of downstream tasks. We here prove that feedforward models trained with objectives belonging to the commonly used InfoNCE family learn to implicitly invert the underlying generative model of the observed data. While the proofs make certain statistical assumptions about the generative model, we observe empirically that our findings hold even if these assumptions are severely violated. Our theory highlights a fundamental connection between contrastive learning, generative modeling, and nonlinear independent component analysis, thereby furthering our understanding of the learned representations as well as providing a theoretical foundation to derive more effective contrastive losses.

Motivation & Objective

• Understand why contrastive representations generalize across downstream tasks.
• Theoretically link contrastive learning with the underlying data-generating process and generative modeling.
• Establish connections to nonlinear independent component analysis to explain representations.
• Provide a theoretical foundation to derive more effective contrastive losses.

Proposed method

• Prove that feedforward models trained with InfoNCE objectives implicitly invert the data-generating process.
• State and rely on statistical assumptions about the generative model.
• Demonstrate the connection between contrastive learning, generative modeling, and nonlinear ICA.
• Provide empirical evidence showing robustness of the theory even when assumptions are violated.
• Outline implications for deriving improved contrastive loss functions.

Experimental results

Research questions

• RQ1 Does InfoNCE-based contrastive learning invert the data-generating process under stated assumptions?
• RQ2 To what extent do the theoretical results hold when the generative-model assumptions are violated?
• RQ3 How are contrastive learning, generative modeling, and nonlinear ICA mathematically connected?
• RQ4 Can the insights guide the design of more effective contrastive loss functions?

Key findings

• Feedforward models trained with InfoNCE objectives implicitly invert the underlying generative model of the observed data.
• The proofs rely on certain statistical assumptions about the generative model.
• Empirically, the findings hold even when those assumptions are severely violated.
• The work establishes a fundamental connection between contrastive learning, generative modeling, and nonlinear ICA.
• The results provide a theoretical foundation to derive more effective contrastive losses.