[Paper Review] The Power of Contrast for Feature Learning: A Theoretical Analysis

Wenlong Ji, Zhun Deng+3 more|arXiv (Cornell University)|Oct 6, 2021

Generative Adversarial Networks and Image Synthesis55 references4 citations

TL;DR

This paper provides a theoretical analysis of contrastive learning under linear representation settings, demonstrating that it outperforms autoencoders and GANs in feature recovery and downstream tasks by effectively reducing noise through contrastive data augmentation. It further explains why supervised contrastive learning improves in-domain performance but can harm transfer learning due to label-induced inductive bias.

ABSTRACT

Contrastive learning has achieved state-of-the-art performance in various self-supervised learning tasks and even outperforms its supervised counterpart. Despite its empirical success, theoretical understanding of the superiority of contrastive learning is still limited. In this paper, under linear representation settings, (i) we provably show that contrastive learning outperforms the standard autoencoders and generative adversarial networks, two classical generative unsupervised learning methods, for both feature recovery and in-domain downstream tasks; (ii) we also illustrate the impact of labeled data in supervised contrastive learning. This provides theoretical support for recent findings that contrastive learning with labels improves the performance of learned representations in the in-domain downstream task, but it can harm the performance in transfer learning. We verify our theory with numerical experiments.

Motivation & Objective

• To theoretically explain why contrastive learning outperforms classical unsupervised methods like autoencoders and GANs in feature learning.
• To analyze the impact of label information in contrastive learning, particularly its dual role in enhancing in-domain performance while potentially degrading transfer learning.
• To establish a theoretical framework using the spiked covariance model to study representation learning under linear assumptions.
• To justify the superiority of contrastive learning in noise reduction and representation quality via data augmentation.

Proposed method

• Theoretical analysis is conducted under a linear representation setting using the spiked covariance model to model data with a low-rank signal and high-dimensional noise.
• The paper derives bounds on the estimation error of the leading eigenvector of the empirical covariance matrix, which corresponds to the true underlying feature direction.
• It compares contrastive learning to autoencoders and GANs by analyzing the bias and variance of the learned representations under different objectives.
• The analysis incorporates data augmentation to construct positive and negative contrastive pairs, showing how this reduces noise in the learned features.
• Theoretical guarantees are derived using random matrix theory, including eigenvalue perturbation bounds and concentration inequalities for sub-Gaussian matrices.
• The framework is extended to supervised contrastive learning by incorporating class labels into the contrastive objective, analyzing its effect on representation quality and generalization.

Experimental results

Research questions

• RQ1 Why does contrastive learning outperform standard autoencoders and GANs in feature recovery and downstream tasks?
• RQ2 How does data augmentation in contrastive learning contribute to noise reduction and improved representation learning?
• RQ3 What is the theoretical impact of label information in supervised contrastive learning on in-domain performance versus transfer learning?
• RQ4 Why does supervised contrastive learning improve in-domain accuracy but potentially harm transfer learning performance?
• RQ5 Under what conditions does contrastive learning achieve better representation learning than generative unsupervised methods?

Key findings

• Contrastive learning provably outperforms autoencoders and GANs in feature recovery by reducing noise through contrastive data augmentation, especially in high-dimensional settings.
• The method achieves a Frobenius norm error bound of order $\sqrt{d/n} \sigma_{(1)}$ for the estimated representation, showing consistent convergence as sample size increases.
• Supervised contrastive learning improves in-domain regression and classification performance by leveraging label information to refine the representation space.
• However, the same label-induced inductive bias can degrade performance in transfer learning, explaining empirical observations of limited or negative transfer gains.
• Theoretical analysis confirms that contrastive learning effectively separates signal from noise by constructing informative negative samples via data augmentation.
• The results are validated numerically, showing alignment between theoretical predictions and empirical performance across varying data dimensions and sample sizes.