[论文解读] Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere

Tongzhou Wang, Phillip Isola|arXiv (Cornell University)|May 20, 2020

Domain Adaptation and Few-Shot Learning被引用 512

一句话总结

这篇论文定义了两个度量——对齐与均匀性，用于在超球面上评估表征，并证明对比损失在渐近意义上优化它们；它还显示直接优化这些度量在下游性能上表现强劲，有时甚至超越标准对比方法。

ABSTRACT

Contrastive representation learning has been outstandingly successful in practice. In this work, we identify two key properties related to the contrastive loss: (1) alignment (closeness) of features from positive pairs, and (2) uniformity of the induced distribution of the (normalized) features on the hypersphere. We prove that, asymptotically, the contrastive loss optimizes these properties, and analyze their positive effects on downstream tasks. Empirically, we introduce an optimizable metric to quantify each property. Extensive experiments on standard vision and language datasets confirm the strong agreement between both metrics and downstream task performance. Remarkably, directly optimizing for these two metrics leads to representations with comparable or better performance at downstream tasks than contrastive learning. Project Page: https://tongzhouwang.info/hypersphere Code: https://github.com/SsnL/align_uniform , https://github.com/SsnL/moco_align_uniform

研究动机与目标

• Motivate and formalize two key properties of contrastive representations: alignment of positive pairs and uniformity on the hypersphere.
• Propose computable metrics for alignment and uniformity with theoretical grounding.
• Show asymptotic convergence of the contrastive loss to alignment and uniformity objectives.
• Empirically validate that aligning and uniforming encodings correlates with and improves downstream tasks.
• Evaluate whether directly optimizing the two metrics can match or surpass traditional contrastive learning in practice.

提出的方法

• Model representations as unit-norm features on the hypersphere via normalization.
• Define alignment loss as the expected distance between positive pair features.
• Define uniformity loss via the logarithm of the average Gaussian potential between features on the hypersphere.
• Prove that as the number of negatives grows, the contrastive loss converges to a form that optimizes alignment and uniformity.
• Connect the uniformity objective to minimizing a Gaussian potential and to entropy/MI interpretations.
• Provide practical PyTorch implementations of the two metrics and evaluate across multiple datasets and baselines.

实验结果

研究问题

• RQ1 Do alignment and uniformity adequately capture quality aspects of representations produced by contrastive learning?
• RQ2 Does the contrastive loss asymptotically optimize alignment and uniformity on the unit hypersphere?
• RQ3 Can directly optimizing alignment and uniformity yield representations that match or exceed those obtained by standard contrastive learning in downstream tasks?
• RQ4 How do these metrics correlate with downstream task performance across vision and language benchmarks?

主要发现

• Contrastive representations exhibit strong alignment (low positive-pair distances) and uniformity (near-uniform distribution on the hypersphere).
• As negative samples grow, the contrastive loss converges to a form that incentivizes alignment and, when perfect uniformity exists, corresponds to the uniform distribution on the hypersphere.
• The proposed alignment and uniformity metrics strongly agree with downstream task performance across benchmarks.
• Directly optimizing for alignment and uniformity yields competitive or superior downstream performance compared with conventional contrastive learning in several settings.
• Encoders optimized with only alignment and uniformity losses outperform those trained with the standard contrastive objective in reported experiments.
• There is causal evidence that improving both alignment and uniformity improves downstream task accuracy.