[Paper Review] Parameter Re-Initialization through Cyclical Batch Size Schedules

Norman Mu, Zhewei Yao+3 more|arXiv (Cornell University)|Dec 4, 2018

Adversarial Robustness in Machine Learning26 references5 citations

TL;DR

This paper proposes cyclical batch size (CBS) schedules to improve neural network training by re-initializing weights through controlled noise injection via alternating large and small batch sizes. The method improves language modeling perplexity by up to 7.91 and reduces training iterations by up to 61%, while enabling efficient ensembling and adversarial training with minimal additional cost.

ABSTRACT

Optimal parameter initialization remains a crucial problem for neural network training. A poor weight initialization may take longer to train and/or converge to sub-optimal solutions. Here, we propose a method of weight re-initialization by repeated annealing and injection of noise in the training process. We implement this through a cyclical batch size schedule motivated by a Bayesian perspective of neural network training. We evaluate our methods through extensive experiments on tasks in language modeling, natural language inference, and image classification. We demonstrate the ability of our method to improve language modeling performance by up to 7.91 perplexity and reduce training iterations by up to $61\%$, in addition to its flexibility in enabling snapshot ensembling and use with adversarial training.

Motivation & Objective

• To address the challenge of sub-optimal weight initialization in deep neural networks, which can lead to poor convergence and generalization.
• To explore adaptive initialization strategies that dynamically adjust noise levels during training using batch size as a control knob.
• To improve training efficiency and model performance across diverse tasks such as language modeling, NLP, and image classification.
• To enable flexible integration with advanced techniques like snapshot ensembling and adversarial training without additional training cost.
• To provide a Bayesian-inspired framework for dynamic re-initialization that enhances generalization and escapes poor local minima.

Proposed method

• The method uses cyclical batch size schedules to modulate the noise level in stochastic gradient descent (SGD), emulating a Bayesian adaptive prior through controlled temperature variation.
• Noise in SGD is governed by the ratio of learning rate to batch size; by cycling batch size, the method dynamically alters the noise scale without changing the learning rate.
• The approach is grounded in the Euler-Maruyama discretization of a stochastic differential equation, where batch size acts as a temperature knob to control exploration.
• Different cyclical schedules (e.g., CBS-10-3, CBS-15-2) are evaluated across architectures like LSTMs, ResNets, and WResNets on tasks including language modeling and image classification.
• The method enables snapshot ensembling by saving models at cycle peaks, achieving ensemble performance at no extra training cost.
• It is combined with adversarial regularization to further improve robustness and accuracy, particularly in image classification.

Experimental results

Research questions

• RQ1 Can cyclical batch size scheduling improve generalization performance in neural network training compared to fixed batch size baselines?
• RQ2 To what extent can CBS schedules reduce training iterations while maintaining or improving model accuracy?
• RQ3 How does CBS perform in scenarios with poor or sub-optimal weight initialization?
• RQ4 Can CBS be effectively combined with snapshot ensembling and adversarial training to enhance model performance?
• RQ5 Does the Bayesian interpretation of CBS as adaptive prior refinement lead to measurable improvements in loss and generalization?

Key findings

• CBS schedules reduced training iterations by up to 61% on image classification tasks, significantly decreasing runtime.
• In language modeling, CBS improved performance by up to 7.91 perplexity points compared to baseline models.
• For natural language inference and image classification, CBS led to minor but consistent accuracy improvements.
• Snapshot ensembling using models from different CBS cycles improved image classification accuracy to 76.401% on ImageNet, up from 75.336% with baseline training.
• Combining CBS with adversarial training yielded an additional 0.26% accuracy gain on ImageNet.
• CBS outperformed baseline initialization even with poor initial weights, demonstrating robustness to sub-optimal initialization.