[Paper Review] Predicting What You Already Know Helps: Provable Self-Supervised Learning

Jason D. Lee, Qi Lei+2 more|arXiv (Cornell University)|Aug 3, 2020

Domain Adaptation and Few-Shot Learning69 references51 citations

TL;DR

The paper provides a theoretical framework showing that reconstruction-based self-supervised learning under approximate conditional independence yields good downstream linear predictors with reduced labeled data, and extends to nonlinear CCA settings like SimSiam.

ABSTRACT

Self-supervised representation learning solves auxiliary prediction tasks (known as pretext tasks) without requiring labeled data to learn useful semantic representations. These pretext tasks are created solely using the input features, such as predicting a missing image patch, recovering the color channels of an image from context, or predicting missing words in text; yet predicting this extit{known} information helps in learning representations effective for downstream prediction tasks. We posit a mechanism exploiting the statistical connections between certain {\em reconstruction-based} pretext tasks that guarantee to learn a good representation. Formally, we quantify how the approximate independence between the components of the pretext task (conditional on the label and latent variables) allows us to learn representations that can solve the downstream task by just training a linear layer on top of the learned representation. We prove the linear layer yields small approximation error even for complex ground truth function class and will drastically reduce labeled sample complexity. Next, we show a simple modification of our method leads to nonlinear CCA, analogous to the popular SimSiam algorithm, and show similar guarantees for nonlinear CCA.

Motivation & Objective

• Motivate and formalize why reconstruction-based self-supervised tasks help downstream prediction.
• Introduce approximate conditional independence (ACI) as a key assumption linking pretext and downstream tasks.
• Provide generalization guarantees showing small representation and estimation errors under ACI.
• Instantiate the theory in topic modeling and connect to nonlinear CCA variants like SimSiam.
• Demonstrate through simulations and real data that SSL reduces labeled data requirements while maintaining performance.

Proposed method

• Define a two-step SSL framework: learn a representation by predicting X2 from X1, then train a linear predictor on Y using the learned representation.
• Derive a closed-form solution for the optimal pretext representation under a linear function space and show Y is linear in the learned representation under conditional independence.
• Establish generalization bounds showing small excess risk with labeled data scaling as O(k/n2) under CI, extended to ACI with latent variables (epsilon_CI, epsilon_pre).
• Extend the analysis to a universal function class (or linear feature maps) and relate representation quality to estimation and approximation errors.
• Connect the SSL objective to a nonlinear CCA/SIM-Siam style objective and provide analogous guarantees.
• Illustrate with topic modeling as a concrete instantiation and discuss how ACI manifests in that setting.

Experimental results

Research questions

• RQ1 Under what statistical conditions do reconstruction-based pretext tasks yield representations that enable accurate downstream predictions with linear classifiers?
• RQ2 How does approximate conditional independence (ACI) with latent variables affect sample complexity and generalization guarantees for SSL?
• RQ3 Can the theory be extended to nonlinear view contrastive-like methods such as SIM-Siam, and what guarantees hold?
• RQ4 How can the framework be instantiated in topic models and other generative settings to quantify SSL benefits?
• RQ5 What are the roles and magnitudes of epsilon_CI and epsilon_pre in the downstream risk bound and how do they influence data requirements?

Key findings

• Under conditional independence X1 ⟂ X2 | Y, the optimal pretext representation psi* = E[X2 | X1] makes the downstream predictor linear in psi*, giving zero approximation error in f* with respect to psi*.
• With psi* and mild assumptions, the downstream excess risk scales as O~(k/n2) for labeled samples, implying reduced labeled data requirements.
• Replacing exact CI with approximate CI (ACI) still yields finite-sample excess risk bounded by a sum of estimation and approximation terms, allowing n2 = O(d2) labeled samples when epsilon_CI and epsilon_pre are small.
• For linear feature maps, the optimal psi* is a linear transform of phi1, and under CI, the representation preserves approximation error while improving sample efficiency.
• A topic-model instantiation demonstrates CIs leading to zero epsilon_CI and a downstream predictor Y that is linear in the learned representation, with bounds depending on the topic covariance and condition number.
• The approach extends to nonlinear CCA-like objectives (e.g., SIM-Siam) with corresponding guarantees, linking SSL reconstruction to two-view representation learning.