[Paper Review] On Warm-Starting Neural Network Training

Jordan T. Ash, Ryan P. Adams|arXiv (Cornell University)|Oct 18, 2019

Machine Learning and Data Classification51 references56 citations

TL;DR

The paper shows that warm-starting neural networks in sequential data settings harms generalization, and introduces a simple shrink-then-perturb trick to close the generalization gap while retaining faster training.

ABSTRACT

In many real-world deployments of machine learning systems, data arrive piecemeal. These learning scenarios may be passive, where data arrive incrementally due to structural properties of the problem (e.g., daily financial data) or active, where samples are selected according to a measure of their quality (e.g., experimental design). In both of these cases, we are building a sequence of models that incorporate an increasing amount of data. We would like each of these models in the sequence to be performant and take advantage of all the data that are available to that point. Conventional intuition suggests that when solving a sequence of related optimization problems of this form, it should be possible to initialize using the solution of the previous iterate -- to "warm start" the optimization rather than initialize from scratch -- and see reductions in wall-clock time. However, in practice this warm-starting seems to yield poorer generalization performance than models that have fresh random initializations, even though the final training losses are similar. While it appears that some hyperparameter settings allow a practitioner to close this generalization gap, they seem to only do so in regimes that damage the wall-clock gains of the warm start. Nevertheless, it is highly desirable to be able to warm-start neural network training, as it would dramatically reduce the resource usage associated with the construction of performant deep learning systems. In this work, we take a closer look at this empirical phenomenon and try to understand when and how it occurs. We also provide a surprisingly simple trick that overcomes this pathology in several important situations, and present experiments that elucidate some of its properties.

Motivation & Objective

• Motivate and analyze the generalization gap observed when warm-starting neural networks in sequential data-update scenarios.
• Quantify how the generalization gap manifests across architectures, optimizers, and datasets.
• Evaluate conventional initialization/regularization approaches in the warm-start context.
• Propose and validate a simple corrective technique to mitigate the warm-start generalization gap.
• Explore implications for online learning and pre-training in data-rich regimes.

Proposed method

• Empirical comparison of warm-started versus randomly initialized networks across architectures (ResNet-18, MLP, Logistic Regression) and datasets (CIFAR-10, CIFAR-100, SVHN).
• Online/batch updating experiments where data arrive in chunks and models are updated with previous parameters vs. fresh starts.
• Systematic hyperparameter sweeps for batch size and learning rate to assess potential remedies.
• Regularization experiments to test if L2, confidence penalties, or adversarial training mitigate the gap.
• Introduction and evaluation of the shrink, perturb, repeat trick with controlled shrinking factor lambda and noise sigma to balance gradients.
• Analysis of gradient magnitudes to understand the mechanism behind the trick.

Experimental results

Research questions

• RQ1 Does warm-starting degrade generalization compared to random initialization when updating neural networks with new data?
• RQ2 How does the generalization gap vary with model type, dataset difficulty, and optimizer?
• RQ3 Can conventional remedies (regularization, hyperparameter tuning) resolve the gap in warm-started training?
• RQ4 Can a simple initialization modification (shrink-perturb) restore generalization and preserve faster training in sequential updates?
• RQ5 What is the interaction between pre-training transfer and warm-starting in data-rich regimes?

Key findings

• Warm-started neural networks consistently exhibit worse generalization than randomly initialized counterparts, even when training accuracy is similar.
• The generalization gap is more pronounced on CIFAR-10 than SVHN; logistic regression (convex loss) is less affected.
• Neither changing batch size nor learning rate fully resolves the gap; in some cases warm-started models forget previous learning while aligned with faster training.
• A simple shrink-perturb trick (theta^t = lambda theta^{t-1} + p^t, p^t ~ N(0, sigma^2)) can close the gap and speed up online training, with lambda around 0.6 often matching random init performance.
• Shrinking preserves the learned hypothesis to a degree, balances gradient contributions between old and new data, and, when combined with perturbation, improves both training efficiency and generalization compared to pure warm-starts.
• Pre-training transfers from unrelated data can hurt performance in data-rich second tasks; shrink-perturb initialization adapts across transfer scenarios, performing at least as well as warm-starts or scratch training depending on data availability.