[论文解读] Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter|arXiv (Cornell University)|Nov 14, 2017

Sparse and Compressive Sensing Techniques被引用 9,013

一句话总结

论文表明 L2 正则化对自适应优化器如 Adam 并不等价于权重衰减，提出解耦权重衰减（AdamW/SGDW），并展示了提升泛化和更易超参数调优。

ABSTRACT

L$_2$ regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is \emph{not} the case for adaptive gradient algorithms, such as Adam. While common implementations of these algorithms employ L$_2$ regularization (often calling it "weight decay" in what may be misleading due to the inequivalence we expose), we propose a simple modification to recover the original formulation of weight decay regularization by \emph{decoupling} the weight decay from the optimization steps taken w.r.t. the loss function. We provide empirical evidence that our proposed modification (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam's generalization performance, allowing it to compete with SGD with momentum on image classification datasets (on which it was previously typically outperformed by the latter). Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

研究动机与目标

• Motivate the study: adaptive gradient methods generalize poorly compared to SGD with momentum on image tasks.
• Show that L2 regularization is inequivalent to weight decay for adaptive optimizers.
• Propose decoupled weight decay as a fix and analyze its impact on generalization and hyperparameter tuning.

提出的方法

• Define weight decay as a decoupled step from gradient-based updates (Algorithm 1 SGDW).
• Extend the decoupling idea to Adam to form AdamW (Algorithm 2).
• Provide theoretical propositions showing inequivalence of L2 and weight decay for adaptive methods (Propositions 2 and 3).
• Justify decoupled weight decay via Bayesian filtering perspective (Aitchison 2018).
• Empirically compare Adam, AdamW, SGD, and SGDW on CIFAR-10, CIFAR-100, and ImageNet32x32 with various learning rate schedules.

实验结果

研究问题

• RQ1 Does L2 regularization equate to weight decay for adaptive gradient methods like Adam?
• RQ2 Does decoupling weight decay from gradient updates improve generalization for Adam?
• RQ3 How do learning rate schedules interact with decoupled weight decay in SGD/Adam?
• RQ4 Is hyperparameter tuning easier when using decoupled weight decay?

主要发现

• L2 regularization is not equivalent to weight decay for adaptive gradient methods like Adam.
• AdamW with decoupled weight decay yields substantially better generalization than Adam with L2 regularization, achieving around 15% relative improvement in test error on tested datasets.
• Decoupling weight decay from the gradient update decouples the learning rate and weight decay hyperparameters, making hyperparameter tuning more separable.
• Cosine annealing and warm restarts enhance the performance of AdamW and SGDW, reducing the gap to SGD with momentum.
• AdamW reports better generalization across CIFAR-10, CIFAR-100, and ImageNet32x32, and AdamWR improves anytime performance significantly.