[Paper Review] Faster Neural Network Training with Data Echoing

Dami Choi, Alexandre Passos+2 more|arXiv (Cornell University)|Jul 12, 2019

Neural Networks and Applications24 references33 citations

TL;DR

The paper introduces data echoing, a method that reuses outputs from upstream data-processing stages to reclaim idle accelerator capacity, reducing upstream work and speeding up training while preserving final performance. It achieves notable wallclock speedups (e.g., 3.25x for ResNet-50 on ImageNet when reading data over a network).

ABSTRACT

In the twilight of Moore's law, GPUs and other specialized hardware accelerators have dramatically sped up neural network training. However, earlier stages of the training pipeline, such as disk I/O and data preprocessing, do not run on accelerators. As accelerators continue to improve, these earlier stages will increasingly become the bottleneck. In this paper, we introduce "data echoing," which reduces the total computation used by earlier pipeline stages and speeds up training whenever computation upstream from accelerators dominates the training time. Data echoing reuses (or "echoes") intermediate outputs from earlier pipeline stages in order to reclaim idle capacity. We investigate the behavior of different data echoing algorithms on various workloads, for various amounts of echoing, and for various batch sizes. We find that in all settings, at least one data echoing algorithm can match the baseline's predictive performance using less upstream computation. We measured a factor of 3.25 decrease in wall-clock time for ResNet-50 on ImageNet when reading training data over a network.

Motivation & Objective

• Motivate reducing upstream data-processing computation when accelerators are the bottleneck in NN training.
• Propose data echoing as a simple, cheap way to reclaim idle accelerator capacity by reusing data from previous pipeline stages.
• Systematically evaluate how different echoing strategies affect training efficiency and predictive performance across diverse workloads.

Proposed method

• Insert an echoing stage after a bottleneck pipeline stage to repeat outputs from the previous stage.
• Formalize echoing factor e and derive the one-upstream-plus-e-downstream-step time as max(t_upstream, e * t_downstream).
• Compare example echoing and batch echoing variants and analyze effects of shuffling and augmentation on performance.
• Tune hyperparameters via quasi-random search to reach target out-of-sample metrics under fixed fresh-example budgets.
• Evaluate across Transformer (LM1B, Common Crawl), ResNet (CIFAR-10, ImageNet), and SSD (COCO) with varying batch sizes.

Experimental results

Research questions

• RQ1 Can data echoing reduce the number of fresh training examples required to reach a target out-of-sample performance across different models and datasets?
• RQ2 How does the insertion point of data echoing affect performance and required upstream/downstream work?
• RQ3 What is the impact of echoing factor, batch size, and shuffling on predictive performance and walltime?
• RQ4 Does data echoing maintain final predictive performance compared to well-tuned baselines?
• RQ5 Under upstream-dominated pipelines, how much walltime speedup can data echoing provide?

Key findings

• Data echoing reduces the number of fresh examples required to reach target out-of-sample performance in all but one case (batch echoing on ResNet-50).
• Earlier insertion points (before augmentation) generally yield fewer fresh examples than later points.
• Echoing can yield walltime speedups proportional to the reduction in fresh examples, with up to 3.25x speedup observed for ResNet-50 on ImageNet when reading data over a network.
• The usefulness of echoing depends on the echoing factor and batch size; feasible factors up to 4–8 for LM1B (1024 batch) and even larger for 4096 batch, with diminishing returns beyond a threshold.
• More shuffling of echoed data improves performance for both example and batch echoing.
• All echoing variants achieved at least the same final out-of-sample performance as the baseline under fixed budgets of fresh data.