[Paper Review] Accelerating Deep Learning by Focusing on the Biggest Losers

Angela H. Jiang, Daniel Wong+9 more|arXiv (Cornell University)|Oct 2, 2019

Advanced Neural Network Applications27 references36 citations

TL;DR

Selective-Backprop prioritizes high-loss training examples to skip costly backpropagation, accelerating deep learning; Stale-SB further reduces selection overhead, achieving up to 3.5x faster training compared to standard SGD.

ABSTRACT

This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update parameters, or to skip immediately to the next example. By reducing the number of computationally-expensive backpropagation steps performed, Selective-Backprop accelerates training. Evaluation on CIFAR10, CIFAR100, and SVHN, across a variety of modern image models, shows that Selective-Backprop converges to target error rates up to 3.5x faster than with standard SGD and between 1.02--1.8x faster than a state-of-the-art importance sampling approach. Further acceleration of 26% can be achieved by using stale forward pass results for selection, thus also skipping forward passes of low priority examples.

Motivation & Objective

• Motivate the need to accelerate DNN training by focusing on informative (hard) examples rather than treating all samples equally.
• Propose a lightweight, loss-based sampling technique to reduce backpropagation work without sacrificing accuracy.
• Evaluate the approach across multiple datasets and architectures to quantify speedups and robustness.

Proposed method

• Introduce Selective-Backprop (SB) that uses forward-pass losses as a proxy for gradient contribution to decide whether to include an example in a backward pass.
• Compute a selection probability P(L) as a monotone function of the current loss percentile (CDF-based) to bias sampling toward high-loss examples.
• Use a running window of recent losses to approximate the current loss distribution when computing P(L).
• Optionally employ Stale-SB to reuse losses from previous selection passes (every nth epoch) to reduce forward-pass cost.
• Provide optional optimizations for reducing selection overhead, such as loss re-use and forward-pass acceleration ideas.
• SB is designed to work with standard SGD and variants (Adam, RMSprop, etc.) without changing hyperparameters.

Experimental results

Research questions

• RQ1 Can selective backpropagation reduce training time to target accuracy without significantly harming final model performance?
• RQ2 How does SB compare to state-of-the-art online importance sampling in terms of speedups and final accuracy?
• RQ3 What is the impact of leveraging stale forward-pass results (Stale-SB) on training efficiency and accuracy?
• RQ4 How robust is SB to label noise and dataset redundancy across common image classification benchmarks?
• RQ5 What are practical deployment considerations and Pareto-optimal configurations for SB across datasets and models?

Key findings

• SB reduces backward-pass computations by skipping low-loss examples, achieving up to 3.5x speedups to target error on CIFAR10/100 and SVHN across several models.
• Stale-SB, reusing forward-pass losses for multiple epochs, yields about 26% additional speedup with minimal accuracy loss.
• SB generally outperforms a state-of-the-art online importance sampling method (Katharopoulos & Fleuret 2018) by 1.02–1.8x in reaching target accuracy.
• Across Pareto-optimal configurations, SB and Stale-SB account for a large share of optimal trade-offs, with SB often providing the best speed-accuracy balance for CIFAR10, CIFAR100, and SVHN.
• SB is robust to modest label noise and can accelerate learning on clean datasets; higher selectivity increases speed but may raise final error.