[Paper Review] Spherical Perspective on Learning with Batch Norm.

Simon Roburin, Yann de Mont-Marin+4 more|arXiv (Cornell University)|Jun 23, 2020

Stochastic Gradient Optimization Techniques17 references4 citations

TL;DR

This paper introduces a geometric framework for understanding batch normalization (BN) in deep learning by modeling optimization on the L2 unit hypersphere, revealing that SGD with momentum under BN is equivalent to a constrained variant of Adam. The approach enables derivation of effective learning rates and leads to new Adam variants that improve classification accuracy across multiple datasets and architectures.

ABSTRACT

Batch Normalization (BN) is a prominent deep learning technique. In spite of its apparent simplicity, its implications over optimization are yet to be fully understood. In this paper, we study the optimization of neural networks with BN layers from a geometric perspective. We leverage the radial invariance of groups of parameters, such as neurons for multi-layer perceptrons or filters for convolutional neural networks, and translate several popular optimization schemes on the $L_2$ unit hypersphere. This formulation and the associated geometric interpretation sheds new light on the training dynamics and the relation between different optimization schemes. In particular, we use it to derive the effective learning rate of Adam and stochastic gradient descent (SGD) with momentum, and we show that in the presence of BN layers, performing SGD alone is actually equivalent to a variant of Adam constrained to the unit hypersphere. Our analysis also leads us to introduce new variants of Adam. We empirically show, over a variety of datasets and architectures, that they improve accuracy in classification tasks. The complete source code for our experiments is available at: this https URL

Motivation & Objective

• To understand the optimization dynamics of neural networks with batch normalization from a geometric perspective.
• To explain the equivalence between SGD with momentum and a constrained form of Adam under batch normalization.
• To derive the effective learning rates of SGD and Adam in the presence of BN layers.
• To develop new optimization variants based on geometric insights that improve model accuracy.
• To empirically validate the proposed methods across diverse architectures and datasets.

Proposed method

• Modeling parameter groups (e.g., neurons, filters) as points on the L2 unit hypersphere to exploit radial invariance.
• Translating standard optimization schemes—such as SGD and Adam—onto the unit hypersphere to reveal geometric structure.
• Using spherical geometry to derive the effective learning rate for SGD with momentum and Adam under BN.
• Introducing new Adam variants constrained to the unit hypersphere, preserving geometric invariance.
• Formulating optimization dynamics using radial and angular components on the hypersphere to separate scale and direction.
• Validating the proposed methods through extensive experiments on image classification benchmarks.

Experimental results

Research questions

• RQ1 How does batch normalization affect the optimization trajectory of deep networks from a geometric standpoint?
• RQ2 What is the true relationship between SGD with momentum and Adam when batch normalization is applied?
• RQ3 Can radial invariance in parameter groups be leveraged to derive more effective optimization algorithms?
• RQ4 What are the effective learning rates of SGD and Adam under batch normalization, and how do they relate geometrically?
• RQ5 Do new Adam variants derived from spherical optimization improve generalization and accuracy in practice?

Key findings

• SGD with momentum under batch normalization is geometrically equivalent to a constrained variant of Adam on the unit hypersphere.
• The effective learning rate of Adam and SGD with momentum can be analytically derived using spherical geometry.
• New Adam variants derived from the spherical formulation achieve higher classification accuracy on multiple datasets and architectures.
• The proposed methods improve performance without increasing model complexity or computational cost.
• Empirical results confirm consistent accuracy gains across diverse network architectures and benchmark datasets.
• The geometric framework provides a unifying perspective on optimization dynamics in BN-regularized networks.