[Paper Review] BYOL works even without batch statistics

Pierre H. Richemond, Jean-Bastien Grill+9 more|arXiv (Cornell University)|Oct 20, 2020

Domain Adaptation and Few-Shot Learning26 references20 citations

TL;DR

This paper demonstrates that Batch Normalization (BN) is not essential for Bootstrap Your Own Latent (BYOL) to learn effective representations, as BYOL achieves 73.9% top-1 accuracy on ImageNet using group normalization and weight standardization—without relying on batch statistics. The results refute the hypothesis that BN's implicit contrastive effect prevents representation collapse.

ABSTRACT

Bootstrap Your Own Latent (BYOL) is a self-supervised learning approach for image representation. From an augmented view of an image, BYOL trains an online network to predict a target network representation of a different augmented view of the same image. Unlike contrastive methods, BYOL does not explicitly use a repulsion term built from negative pairs in its training objective. Yet, it avoids collapse to a trivial, constant representation. Thus, it has recently been hypothesized that batch normalization (BN) is critical to prevent collapse in BYOL. Indeed, BN flows gradients across batch elements, and could leak information about negative views in the batch, which could act as an implicit negative (contrastive) term. However, we experimentally show that replacing BN with a batch-independent normalization scheme (namely, a combination of group normalization and weight standardization) achieves performance comparable to vanilla BYOL ($73.9\%$ vs. $74.3\%$ top-1 accuracy under the linear evaluation protocol on ImageNet with ResNet-$50$). Our finding disproves the hypothesis that the use of batch statistics is a crucial ingredient for BYOL to learn useful representations.

Motivation & Objective

• To investigate whether batch normalization (BN) is essential for BYOL to avoid representation collapse and achieve high performance.
• To test the hypothesis that BN provides an implicit contrastive term through gradient flow across batch elements.
• To evaluate whether alternative normalization schemes can replace BN while maintaining competitive performance in BYOL.
• To determine whether proper weight initialization alone can compensate for the absence of BN in BYOL.
• To assess whether the performance gap between BN and BN-free setups can be closed without using batch statistics.

Proposed method

• Replaced all batch normalization (BN) layers in BYOL with group normalization (GN) and weight standardization (WS), eliminating batch statistics during training.
• Used a modified weight initialization scheme that preserves the scaling effect of BN by initializing scale and shift parameters based on initial batch statistics, but without computing them during training.
• Trained the model with adjusted hyperparameters: increased warmup epochs to 50, adjusted weight decay to 3×10⁻⁸, base learning rate to 0.24, and target update rate to 0.999.
• Evaluated performance using linear evaluation protocol on ImageNet after 1000 epochs.
• Compared results across four variants: vanilla BN, no normalization, modified initialization without BN, and GN+WS without BN.
• Conducted ablation studies to isolate the role of BN in optimization stability and representation quality.

Experimental results

Research questions

• RQ1 Is batch normalization (BN) necessary for BYOL to avoid representation collapse?
• RQ2 Does BN provide an implicit contrastive signal through batch statistics that prevents collapse?
• RQ3 Can alternative normalization schemes like group normalization and weight standardization replace BN in BYOL without performance loss?
• RQ4 To what extent can proper weight initialization compensate for the absence of BN in BYOL?
• RQ5 Can BYOL achieve high performance without relying on batch statistics in the training process?

Key findings

• BYOL achieves 65.7% top-1 accuracy on ImageNet under linear evaluation when BN is completely removed and replaced with a modified weight initialization scheme, demonstrating that collapse is avoidable without BN.
• The performance gap between BN and BN-free training is significantly reduced when replacing BN with group normalization and weight standardization, achieving 73.9% top-1 accuracy—very close to the 74.3% of vanilla BYOL.
• The GN+WS variant does not compute batch statistics during training, proving that an implicit contrastive term via batch statistics is not required for high performance.
• Removing all BN layers entirely results in a collapse to near-random performance (0.1% top-1 accuracy), confirming that BN’s role is not merely architectural but critical for training stability.
• The study confirms that BN’s primary function in BYOL is to stabilize training and improve optimization dynamics, not to provide an implicit contrastive signal.
• The results refute the hypothesis that BN’s gradient flow across batch elements acts as an implicit negative term, as performance remains high even without such gradient leakage.