[论文解读] Measuring the Effects of Data Parallelism on Neural Network Training

Christopher J. Shallue, Jaehoon Lee+4 more|arXiv (Cornell University)|Nov 8, 2018

Adversarial Robustness in Machine Learning参考文献 58被引用 163

一句话总结

该论文通过实验表征批量大小（数据并行）如何影响达到目标的样本外误差所需的训练步数，覆盖多样化的工作负载，发现变化很大；较大批量并不一定导致样本外性能下降，同时强调在不同阶段的潜在收益以及元参数调优的作用。

ABSTRACT

Recent hardware developments have dramatically increased the scale of data parallelism available for neural network training. Among the simplest ways to harness next-generation hardware is to increase the batch size in standard mini-batch neural network training algorithms. In this work, we aim to experimentally characterize the effects of increasing the batch size on training time, as measured by the number of steps necessary to reach a goal out-of-sample error. We study how this relationship varies with the training algorithm, model, and data set, and find extremely large variation between workloads. Along the way, we show that disagreements in the literature on how batch size affects model quality can largely be explained by differences in metaparameter tuning and compute budgets at different batch sizes. We find no evidence that larger batch sizes degrade out-of-sample performance. Finally, we discuss the implications of our results on efforts to train neural networks much faster in the future. Our experimental data is publicly available as a database of 71,638,836 loss measurements taken over the course of training for 168,160 individual models across 35 workloads.

研究动机与目标

• Quantify how batch size relates to the number of training steps required to achieve a target out-of-sample error.
• Identify factors that govern the batch size–training steps relationship across models, data sets, and training algorithms.
• Assess whether larger batch sizes incur a cost in out-of-sample performance under realistic workloads.
• Investigate how metaparameters (learning rate, momentum, schedules) should be tuned across batch sizes and explain inconsistencies in prior literature.

提出的方法

• Study synchronous data-parallel mini-batch SGD variants (SGD, SGD with momentum, and Nesterov momentum).
• Experiment across six model families, three training algorithms, and seven data sets to characterize batch-size effects.
• Independently tune learning rate, momentum, and learning-rate schedules for each batch size rather than assuming fixed heuristics.
• Analyze training cost in terms of the number of training steps and report a public data resource with 71,638,836 loss measurements over 168,160 models.
• Provide a reproducible experimental protocol and release the dataset to facilitate replication of plots and results.

实验结果

研究问题

• RQ1 What is the relationship between batch size and the number of training steps to reach a given out-of-sample error?
• RQ2 What factors govern this batch-size–training steps relationship across workloads (model, data set, algorithm)?
• RQ3 Do large batch sizes incur a cost in out-of-sample error across realistic workloads?
• RQ4 How do metaparameters need to be tuned as batch size varies, and do simple scaling rules hold across problems?

主要发现

• The batch-size–training-steps relationship follows a common form across workloads: initial proportional decrease in steps with batch size, followed by diminishing returns, and eventually no improvement beyond a maximum useful batch size.
• The maximum useful batch size varies significantly with workload and depends on model and training algorithm properties; SGD with momentum (and Nesterov momentum) can leverage larger batches than plain SGD, and some models tolerate much larger batch sizes than others.
• Optimal training metaparameters do not follow simple, universal relationships with batch size; linear learning-rate scaling and other heuristics do not hold uniformly across problems and batch sizes.
• Differences in prior literature can be explained by varying computational budgets and metaparameter tuning procedures; there is no evidence that increasing batch size necessarily degrades out-of-sample performance, though larger batch sizes may require additional regularization.