[Paper Review] Self-supervised Representation Learning From Random Data Projectors

Yi Sui, Tongzi Wu+6 more|arXiv (Cornell University)|Oct 11, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes a modality-agnostic self-supervised representation learning method called Learning from Randomness (LFR), which trains models to reconstruct random data projections instead of relying on data augmentations or masking. The approach achieves state-of-the-art performance across diverse modalities—including vision, tabular, and time-series data—without requiring domain-specific augmentation design or specialized architectures.

ABSTRACT

Self-supervised representation learning~(SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. This paper presents an SSRL approach that can be applied to any data modality and network architecture because it does not rely on augmentations or masking. Specifically, we show that high-quality data representations can be learned by reconstructing random data projections. We evaluate the proposed approach on a wide range of representation learning tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. Due to its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study.

Motivation & Objective

• To address the limitations of augmentation-based self-supervised learning (SSRL) in non-vision or non-NLP domains where suitable data augmentations are hard to define.
• To overcome the risk of generating unrealistic or unphysical data examples when applying standard augmentations to sensitive data, such as histopathology images or particle physics tabular data.
• To develop a general-purpose SSRL framework that is applicable to any data modality and neural network architecture without requiring architectural constraints or domain-specific augmentation pipelines.
• To demonstrate that learning from random projections can yield high-quality representations comparable to or better than state-of-the-art contrastive and masked autoencoding methods.

Proposed method

• The method trains a neural network encoder to learn representations such that, for a set of randomly generated projection functions $ G = \{g^{(k)}(\mathbf{x})\} $, the model can predict the outputs of these functions from the learned representation $ \mathbf{z} $.
• Each random projector $ g^{(k)} $ is implemented as a small neural network with random, fixed weights, applied to input data $ \mathbf{x} $, producing a scalar or vector output.
• The model is trained via a reconstruction loss that minimizes the difference between the predicted outputs $ h^{(k)}(\mathbf{z}) $ and the true random projections $ g^{(k)}(\mathbf{x}) $, using a multi-head predictor head.
• The training process alternates between updating the encoder $ f_{\theta} $ and the predictors $ H_{\Phi} $, with the predictors trained for multiple epochs per encoder update to improve stability and performance.
• The approach is architecture-agnostic and does not require transformers or specific backbone designs, enabling use with standard convolutional or fully connected networks.
• Hyperparameters such as the number of projectors $ K $, batch size, and latent dimension are tuned empirically, with $ K=6 $, batch size 256, and latent dimension 2048 found to yield optimal performance on benchmark tasks.

Figure 1: Top: H&E stained histopathology images have a characteristic appearance with blue tones indicating cell nuclei, while cytoplasm is stained pink (Chan, 2014 ) . Bottom: Color jitter with the standard settings of (Chen & He, 2021 ) produces unrealistic augmentations with altered meanings. Ch

Experimental results

Research questions

• RQ1 Can high-quality self-supervised representations be learned without relying on data augmentations or masking, particularly in data modalities where such methods are inapplicable or harmful?
• RQ2 How does the performance of the proposed LFR method compare to state-of-the-art contrastive and masked autoencoding baselines across diverse data modalities?
• RQ3 What is the impact of key hyperparameters—such as the number of random projectors, batch size, and predictor training schedule—on the quality of learned representations?
• RQ4 Does the method generalize effectively to real-world applications with strict data constraints, such as medical imaging or physical simulation data?
• RQ5 Can random projections serve as a viable and robust alternative to heuristic pretext tasks in self-supervised representation learning?

Key findings

• The LFR method outperforms multiple state-of-the-art SSRL baselines across a wide range of data modalities, including vision, tabular, and time-series data, demonstrating strong generalization capabilities.
• On the Kvasir dataset for histopathology image classification, the method achieved a linear evaluation accuracy of 94.8% with 6 random projectors, showing strong performance even without domain-specific augmentations.
• The optimal number of random projectors was found to be 6; increasing beyond this point led to performance degradation due to feature redundancy and gradient bias.
• Batch size had minimal impact on performance, with stable results across 64 to 512, though 256 yielded the best accuracy, indicating robustness to batch size variation.
• Increasing the latent dimension improved performance up to 2048, after which gains became negligible, suggesting a saturation point for representation capacity.
• Alternating training with multiple predictor epochs per encoder update improved performance over joint training, indicating that well-optimized predictors enhance representation quality.

Figure 2: Our proposed architecture for learning from randomness. An input $\mathbf{x}$ is encoded by $f_{\theta}$ into a useful representation $\mathbf{z}$ , while also being fed to random projection functions $g^{(k)}$ . Simple, learnable predictor functions $h^{(k)}_{\phi}$ try to match the outpu