[论文解读] An Exponential Learning Rate Schedule for Deep Learning

Zhiyuan Li, Sanjeev Arora|arXiv (Cornell University)|Oct 16, 2019

Advanced Neural Network Applications参考文献 23被引用 44

一句话总结

本文证明，在带有权重衰减和动量的 BN 启用网络中，指数增大的学习率可以模拟标准 BN+SGD 动态，并提供 WD 与指数 LR 之间的严格等价关系，包括多阶段学习率计划。

ABSTRACT

Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN. 1. Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i.e., learning rate increases by some $(1 +α)$ factor in every epoch for some $α>0$. (Precise statement in the paper.) To the best of our knowledge this is the first time such a rate schedule has been successfully used, let alone for highly successful architectures. As expected, such training rapidly blows up network weights, but the net stays well-behaved due to normalization. 2. Mathematical explanation of the success of the above rate schedule: a rigorous proof that it is equivalent to the standard setting of BN + SGD + StandardRate Tuning + Weight Decay + Momentum. This equivalence holds for other normalization layers as well, Group Normalization, LayerNormalization, Instance Norm, etc. 3. A worked-out toy example illustrating the above linkage of hyper-parameters. Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.

研究动机与目标

• Motivate why learning rate schedules interact with Batch Normalization and normalization layers in deep nets.
• Show that an exponential learning rate schedule can mimic the effect of weight decay in SGD with momentum under scale-invariant objectives.
• Provide a formal equivalence between SGD with weight decay and exponential LR schedules across normalization schemes.
• Explain multi-phase and tapered exponential LR schemes and how they relate to standard practice like step decay.
• Demonstrate implications through toy examples and CNN/ResNet experiments.

提出的方法

• Define SGD with momentum and weight decay as in the paper’s Definition 1.2.
• Develop a formal mapping showing GD with WD is equivalent to GD with an exponential LR, via state and map equivalences (Theorem 2.1).
• Extend the equivalence to momentum-enabled SGD (Theorem 2.9).
• Generalize to multi-phase LR schedules and derive a tapered exponential LR (Theorem 2.12).
• Introduce TEXP++ as an LR schedule closely matching Step Decay trajectories (Theorem 2.13).
• Use a toy example to illustrate the WD/BN interplay and nonconvergence when both are used.

实验结果

研究问题

• RQ1 2-5 concrete research questions the paper investigates.

主要发现

• Theorem 1.1 informal states that SGD with scale-invariant objectives and WD can be reformulated as SGD with momentum and an exponential LR schedule, under certain conditions.
• Theorem 2.9 shows GD with WD is equivalent to GD with Exp LR even when momentum is present.
• Theorem 2.12 introduces a tapered-exponential LR schedule (TEXP) that matches Step Decay trajectories with momentum and no WD, via momentum corrections at phase starts.
• Theorem 2.13 presents TEXP++ as an LR scheme that yields the same function-space network sequence as Step Decay with WD, without requiring precise momentum corrections at phase boundaries.
• A toy example demonstrates that BN alone and WD alone can each lead to convergence, but BN+WD together can prevent convergence to a small training error, illustrating their inseparable interaction.
• Experiments on CNNs/ResNets validate the exponential LR concept and show improved or comparable trajectories to traditional schedules in practice.