[Paper Review] Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks

Haoran You, Chaojian Li+7 more|arXiv (Cornell University)|Sep 26, 2019

Advanced Neural Network Applications42 references50 citations

TL;DR

The paper reveals that winning tickets can be identified very early in training (Early-Bird tickets) using low-cost training and mask-distance metrics, and introduces EB Train for substantial energy savings with equal or better accuracy.

ABSTRACT

(Frankle & Carbin, 2019) shows that there exist winning tickets (small but critical subnetworks) for dense, randomly initialized networks, that can be trained alone to achieve comparable accuracies to the latter in a similar number of iterations. However, the identification of these winning tickets still requires the costly train-prune-retrain process, limiting their practical benefits. In this paper, we discover for the first time that the winning tickets can be identified at the very early training stage, which we term as early-bird (EB) tickets, via low-cost training schemes (e.g., early stopping and low-precision training) at large learning rates. Our finding of EB tickets is consistent with recently reported observations that the key connectivity patterns of neural networks emerge early. Furthermore, we propose a mask distance metric that can be used to identify EB tickets with low computational overhead, without needing to know the true winning tickets that emerge after the full training. Finally, we leverage the existence of EB tickets and the proposed mask distance to develop efficient training methods, which are achieved by first identifying EB tickets via low-cost schemes, and then continuing to train merely the EB tickets towards the target accuracy. Experiments based on various deep networks and datasets validate: 1) the existence of EB tickets, and the effectiveness of mask distance in efficiently identifying them; and 2) that the proposed efficient training via EB tickets can achieve up to 4.7x energy savings while maintaining comparable or even better accuracy, demonstrating a promising and easily adopted method for tackling cost-prohibitive deep network training. Code available at https://github.com/RICE-EIC/Early-Bird-Tickets.

Motivation & Objective

• Demonstrate the existence of Early-Bird (EB) tickets that emerge early in training across models and datasets.
• Show that EB tickets can be identified with low-cost training schemes and a practical mask-distance metric.
• Develop EB Train, a training framework that leverages EB tickets to reduce training energy and FLOPs while maintaining accuracy.
• Evaluate EB Train against state-of-the-art pruning-based training methods on CIFAR and ImageNet.
• Provide insights into how high learning rates and low-precision training affect EB ticket emergence.”],
• method3-6 bullet points on the proposed method (key techniques, core equations)

Proposed method

• Define EB tickets as subnetworks that match or exceed dense model accuracy when trained from the pruned mask at t << i.
• Prune dense networks early using channel-wise pruning based on BN scaling factors and a binary mask m.
• Introduce a mask-distance metric, specifically Hamming distance between ticket masks, to detect EB ticket emergence.
• Use a FIFO queue of mask distances to trigger EB ticket identification when recent distances fall below a threshold ε (e.g., 0.1).
• Implement EB Train: (a) search for EB tickets with low-cost training (including high learning rates and 8-bit precision in search), (b) retrain only the EB tickets to target accuracy.
• Compare EB Train variants: FF (full precision search/retrain), re-init, LF (low-precision search, full-precision retrain), LL (low-precision search/retrain).
• Demonstrate energy and FLOPs savings on CIFAR-10/100, ImageNet with PreResNet101, VGG16, ResNet18/50.
• Inherit weights from EB tickets rather than rewinding to initialization, following observed benefits.

Experimental results

Research questions

• RQ1 Do EB tickets consistently exist across popular models and datasets?
• RQ2 Can EB tickets be reliably identified with low-cost training and mask-distance metrics without full training?
• RQ3 Does training only the EB tickets (EB Train) yield comparable or better accuracy with substantial energy/FLOPs savings compared to traditional pruning/retraining?
• RQ4 How do high learning rates and low-precision training affect the emergence and utility of EB tickets?
• RQ5 How does EB Train perform on larger datasets (ImageNet) and architectures (ResNet variants) relative to state-of-the-art baselines?

Key findings

• EB tickets consistently emerge very early (as early as epoch 20 of 160) and can outperform tickets drawn at full training.
• Mask-distance between consecutive epoch tickets stabilizes early; when the max distance in a window falls below ε (0.1), EB tickets are identified.
• EB Train achieves 2.2–2.4x FLOPs reduction over baselines with comparable or better accuracy on CIFAR with PreResNet101/VGG16.
• EB Train with low-precision search/retrain (FP8/8-bit) achieves 5.8–24.6x energy savings and 1.1–5.0x FLOPs savings while maintaining or improving accuracy in many settings.
• On ImageNet (ResNet18/50), EB Train reduces training FLOPs by 51.5–74.0% and training energy by 46.5–70.9%, with accuracy improvements up to +2.34% for some configurations.
• Inheritance of initialization (re-init vs. inherited EB ticket weights) benefits retraining performance, favoring the EB ticket approach over random reinitialization.