[Paper Review] Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methods

Randall Balestriero, Yann LeCun|arXiv (Cornell University)|May 23, 2022

Civil and Geotechnical Engineering Research22 citations

TL;DR

The paper unifies SSL methods (VICReg, SimCLR, BarlowTwins) with spectral embedding techniques, derives closed-form representations and network parameters, and provides principled design guidelines distinguishing global versus local spectral perspectives.

ABSTRACT

Self-Supervised Learning (SSL) surmises that inputs and pairwise positive relationships are enough to learn meaningful representations. Although SSL has recently reached a milestone: outperforming supervised methods in many modalities\dots the theoretical foundations are limited, method-specific, and fail to provide principled design guidelines to practitioners. In this paper, we propose a unifying framework under the helm of spectral manifold learning to address those limitations. Through the course of this study, we will rigorously demonstrate that VICReg, SimCLR, BarlowTwins et al. correspond to eponymous spectral methods such as Laplacian Eigenmaps, Multidimensional Scaling et al. This unification will then allow us to obtain (i) the closed-form optimal representation for each method, (ii) the closed-form optimal network parameters in the linear regime for each method, (iii) the impact of the pairwise relations used during training on each of those quantities and on downstream task performances, and most importantly, (iv) the first theoretical bridge between contrastive and non-contrastive methods towards global and local spectral embedding methods respectively, hinting at the benefits and limitations of each. For example, (i) if the pairwise relation is aligned with the downstream task, any SSL method can be employed successfully and will recover the supervised method, but in the low data regime, VICReg's invariance hyper-parameter should be high; (ii) if the pairwise relation is misaligned with the downstream task, VICReg with small invariance hyper-parameter should be preferred over SimCLR or BarlowTwins.

Motivation & Objective

• Unify self-supervised learning (SSL) methods with spectral embedding theory.
• Derive closed-form optimal representations for SSL losses given a relation matrix G.
• Obtain closed-form optimal network parameters in the linear regime for SSL losses.
• Characterize when SSL solutions align with global vs local spectral embedding methods.
• Provide practical design guidelines for choosing SSL objectives and hyperparameters based on G and downstream tasks.

Proposed method

• Map VICReg, SimCLR, and BarlowTwins to spectral embedding counterparts (Laplacian Eigenmaps, MDS, CCA, LPP, LDA).
• Derive a closed-form optimal representation Z* for SSL losses as a function of the relation matrix G and loss hyperparameters.
• Compute exact optimal linear network parameters for SSL losses in the linear regime.
• Show that SSL methods constrain left singular vectors of Z to align with G’s left singular vectors while not constraining right singular vectors.
• Characterize the role of hyperparameters (e.g., gamma/alpha) in controlling rank and G-information incorporation.
• Provide analytical connections and potential variations of SSL objectives derived from first principles.

Experimental results

Research questions

• RQ1 Can SSL losses be exactly mapped to classical spectral embedding methods?
• RQ2 What is the closed-form optimal representation Z* for VICReg, SimCLR/NNCLR, and BarlowTwins given G?
• RQ3 How do hyperparameters influence rank and information capture of the SSL representations?
• RQ4 Under what conditions do SSL representations align with downstream tasks and supervised counterparts?

Key findings

• VICReg, SimCLR, and BarlowTwins correspond to spectral embedding methods such as Laplacian Eigenmaps, ISOMAP, and CCA under appropriate regimes.
• The optimal SSL representation Z* can be obtained in closed form from a combination of Laplacians, enabling explicit spectral interpretation (Theorem 1 and related results).
• In the linear regime, SSL network parameters W* have closed-form expressions, revealing what input statistics the network emphasizes.
• VICReg can yield a full-rank Z* by tuning loss hyperparameters (gamma/alpha), while SimCLR and BarlowTwins enforce rank(Z)=rank(G), indicating different robustness to G misspecification.
• Contrastive methods align with global spectral embeddings (focusing on preserving global structure), while non-contrastive methods align with local spectral embeddings (focusing on local smoothness).
• The left singular vectors of Z are constrained to match those of G, and, given the right spectrum alignment with Y, any SSL loss can produce an optimal representation for downstream tasks.