[Paper Review] Blockwise Self-Supervised Learning at Scale

Shoaib Ahmed Siddiqui, David Krueger+2 more|arXiv (Cornell University)|Feb 3, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a blockwise self-supervised learning method for training ResNet-50 on ImageNet using Barlow Twins' loss function applied independently to each block of layers, achieving 70.48% top-1 accuracy—only 1.1% below end-to-end backpropagation. The approach trains blocks simultaneously, uses feature expansion pooling, and injects Gaussian noise to improve performance, demonstrating that local learning rules can scale to large models with minimal performance drop.

ABSTRACT

Current state-of-the-art deep networks are all powered by backpropagation. In this paper, we explore alternatives to full backpropagation in the form of blockwise learning rules, leveraging the latest developments in self-supervised learning. We show that a blockwise pretraining procedure consisting of training independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins' loss function at each block performs almost as well as end-to-end backpropagation on ImageNet: a linear probe trained on top of our blockwise pretrained model obtains a top-1 classification accuracy of 70.48%, only 1.1% below the accuracy of an end-to-end pretrained network (71.57% accuracy). We perform extensive experiments to understand the impact of different components within our method and explore a variety of adaptations of self-supervised learning to the blockwise paradigm, building an exhaustive understanding of the critical avenues for scaling local learning rules to large networks, with implications ranging from hardware design to neuroscience.

Motivation & Objective

• To investigate whether blockwise training with local learning rules can achieve performance comparable to end-to-end backpropagation on large-scale datasets like ImageNet.
• To understand the impact of architectural choices such as block order, pooling strategies, and noise injection on blockwise self-supervised learning.
• To explore the feasibility of scaling local learning rules to deep networks, with implications for neuroscience and energy-efficient hardware design.
• To determine whether simultaneous or sequential block training yields better representation learning in a blockwise paradigm.
• To evaluate whether early block pretraining with supervised objectives harms overall network performance in a blockwise self-supervised setup.

Proposed method

• The method applies Barlow Twins’ self-supervised loss function independently to each of the four main blocks of a ResNet-50, training each block in isolation using contrastive representation learning.
• Blocks are trained simultaneously rather than sequentially, preserving inter-block information flow through the feedforward path.
• An expansion pooling strategy increases the feature dimensionality of block outputs before applying the loss, which is critical for successful training.
• Gaussian noise with σ = 0.25 is injected into activations before each block to improve generalization, boosting accuracy by ~0.5%.
• Projector networks are used to map block outputs to a latent space where the Barlow Twins loss is computed, ensuring invariance and information preservation.
• The approach avoids backpropagation across blocks, but retains backpropagation within each block and across its internal layers, including the projector networks.

Experimental results

Research questions

• RQ1 Can blockwise self-supervised learning with local loss functions achieve performance close to end-to-end backpropagation on ImageNet?
• RQ2 Does simultaneous training of blocks outperform sequential training in a blockwise paradigm, and why?
• RQ3 How do different pooling and feature dimensionality strategies affect the performance of blockwise self-supervised learning?
• RQ4 Does pretraining lower blocks with supervised objectives degrade overall network performance in a blockwise self-supervised setup?
• RQ5 Can noise injection and adaptive augmentation strategies improve the representation quality of blockwise-trained models?

Key findings

• The blockwise self-supervised method achieved a linear probe top-1 accuracy of 70.48% on ImageNet, only 1.1% below the 71.57% accuracy of an end-to-end backpropagation-trained ResNet-50.
• Simultaneous training of blocks significantly outperformed sequential training, with a 1.72% accuracy gain, indicating that feedforward path interactions are crucial for learning.
• Injecting Gaussian noise with σ = 0.25 into activations improved performance by approximately 0.5%, while higher noise levels (σ = 0.5) degraded results.
• Expanding the feature dimensionality of block outputs via pooling was essential for successful training, outperforming simple average or max pooling.
• Supervised pretraining of lower blocks reduced overall network performance, suggesting that premature invariance learning in early layers harms higher-level feature learning.
• Merging the first two blocks into a single block increased final accuracy, indicating that excessive block subdivision degrades performance and limits scalability to fully local learning.