[Paper Review] FreezeOut: Accelerate Training by Progressively Freezing Layers

Andrew Brock, Theodore Lim+2 more|arXiv (Cornell University)|Jun 15, 2017

Advanced Neural Network Applications11 references75 citations

TL;DR

FreezeOut speeds up neural network training by progressively freezing hidden layers and excluding them from backpropagation, achieving up to 20% wall-clock speedup with minimal accuracy loss on some architectures.

ABSTRACT

The early layers of a deep neural net have the fewest parameters, but take up the most computation. In this extended abstract, we propose to only train the hidden layers for a set portion of the training run, freezing them out one-by-one and excluding them from the backward pass. Through experiments on CIFAR, we empirically demonstrate that FreezeOut yields savings of up to 20% wall-clock time during training with 3% loss in accuracy for DenseNets, a 20% speedup without loss of accuracy for ResNets, and no improvement for VGG networks. Our code is publicly available at https://github.com/ajbrock/FreezeOut

Motivation & Objective

• Motivate reducing training time by leveraging layers that converge early and have fewer parameters.
• Propose a layer-wise learning-rate schedule that freezes layers progressively during training.
• Evaluate the method across DenseNets, Wide ResNets, and VGG to identify where it yields speedups and where it does not.
• Provide practical defaults and guidelines for applying FreezeOut in common CNN architectures.

Proposed method

• Apply cosine annealing learning rates with no restarts in a layer-wise schedule.
• Freeze the first layer at time t0 and progressively freeze subsequent layers at later times ti.
• Optionally scale the initial learning rates per layer and cube the ti values to bias later layers.
• Compute per-layer learning-rate schedules: ai(t)=0.5*ai(0)*(1+cos(pi*t/ti)).
• Exclude a layer from backward pass once its learning rate decays to zero, lowering per-iteration cost.
• Provide four scheduling variants: linear vs cubic ti progression, unscaled vs scaled learning rates, plus a recommended default configuration.

Experimental results

Research questions

• RQ1 Does progressively freezing layers reduce training time without prohibitive loss in accuracy across common CNN architectures?
• RQ2 Which scheduling variant (linear/cubic, scaled/unscaled) provides the best trade-off between speedup and accuracy?
• RQ3 How does FreezeOut perform on architectures with and without skip connections (DenseNets/ResNets vs VGG)?

Key findings

• Up to 20% wall-clock speedup during training across tested networks.
• DenseNets show up to ~3% increase in test error with FreezeOut under some configurations.
• ResNets achieve about 20% speedup with no loss in accuracy in some settings.
• VGG networks show no improvement from FreezeOut.
• A cubic scheduling with learning-rate scaling is recommended for maximizing speed within ~3% accuracy loss.