[Paper Review] A Theoretical Analysis of Contrastive Unsupervised Representation Learning

Sanjeev Arora, Hrishikesh Khandeparkar+3 more|arXiv (Cornell University)|Feb 25, 2019

Domain Adaptation and Few-Shot Learning24 references258 citations

TL;DR

The paper presents a theoretical framework for contrastive unsupervised representation learning, introduces latent classes to formalize semantic similarity, and proves generalization guarantees for downstream linear classification using mean classifiers, with extensions to multiple negatives and block similarity tested experimentally.

ABSTRACT

Recent empirical works have successfully used unlabeled data to learn feature representations that are broadly useful in downstream classification tasks. Several of these methods are reminiscent of the well-known word2vec embedding algorithm: leveraging availability of pairs of semantically "similar" data points and "negative samples," the learner forces the inner product of representations of similar pairs with each other to be higher on average than with negative samples. The current paper uses the term contrastive learning for such algorithms and presents a theoretical framework for analyzing them by introducing latent classes and hypothesizing that semantically similar points are sampled from the same latent class. This framework allows us to show provable guarantees on the performance of the learned representations on the average classification task that is comprised of a subset of the same set of latent classes. Our generalization bound also shows that learned representations can reduce (labeled) sample complexity on downstream tasks. We conduct controlled experiments in both the text and image domains to support the theory.

Motivation & Objective

• Formalize semantic similarity via latent classes and show downstream tasks comprise a subset of these classes.
• Prove that a representation learned by contrastive unsupervised loss leads to low average supervised loss with a mean classifier.
• Provide generalization bounds for the learned representations based on Rademacher complexity.
• Investigate limitations of negative sampling and propose extensions to leverage larger blocks of similar points.
• Validate the theory with controlled experiments in text and image domains.

Proposed method

• Define similarity as pairs drawn from the same latent class with a class distribution rho.
• Introduce unsupervised contrastive loss L_un using similar and negative samples, and supervised loss L_sup with linear classifiers.
• Show L_sup is bounded by a function of L_un and a generalization term Gen_M (via a Rademacher average).
• Use a mean classifier W^μ whose rows are class means μ_c to connect unsupervised and supervised losses.
• Analyze the impact of class collision (tau) and intraclass deviation s(f) on performance guarantees.
• Extend the framework to k negative samples and to block-based similarity losses that average over blocks of samples.

Experimental results

Research questions

• RQ1 Under what conditions does minimizing the unsupervised contrastive loss yield good supervised (linear) classification performance?
• RQ2 How do class collision (tau) and intraclass variation (s(f)) affect the guarantees provided by contrastive learning?
• RQ3 Can the framework accommodate multiple negative samples and block-based similarity to improve guarantees and practice?
• RQ4 What are the limitations of contrastive learning and can extensions restore competitive guarantees against fully supervised representations?

Key findings

• A surrogate relationship is established: low unsupervised loss implies low supervised performance when averaged over latent classes.
• A bound shows L_sup^μ() can be controlled by L_un^{neq}(f) and intraclass deviation s(f) under broad conditions, with Gen_M capturing finite-sample effects.
• Negative sampling has limitations due to class collisions; the framework quantifies when these hurt and how to mitigate them.
• Using blocks of similar points (instead of pairs) yields tighter bounds and potential empirical improvements.
• The analysis is complemented by controlled experiments in text and image domains supporting the theoretical framework.