[論文レビュー] Training Neural Networks Without Gradients: A Scalable ADMM Approach

Gavin Taylor, Ryan Burmeister+4 more|arXiv (Cornell University)|May 6, 2016

Stochastic Gradient Optimization Techniques参考文献 26被引用数 146

ひとこと要約

勾配降下法なしでニューラルネットを訓練するための ADMM/Bregman ベースの手法を導入し、千コア規模での線形スケーリングと大規模データセットでの頑健な性能を実現します。

ABSTRACT

With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don't scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization sub-steps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly non-convex problems. The method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores.

研究の動機と目的

• 大規模なニューラルネットワークの勾配ベースの訓練の限界を動機づけて対処する。
• 閉形式解を持つ解けるサブステップに訓練を分解する交互最小化フレームワークを提案する。
• 分散設定でのスケーラビリティを実証し、大規模データセットで標準的な勾配ベース法との性能を比較する。
• 実装、初期化、およびパラメータ選択に関する実践的な指針を提供する。
• 理論的解釈とリカレントおよび畳み込みニューラルネットワークへの潜在的拡張を議論する。

提案手法

• W_l を活性化からデカップリングするために、補助変数 z_l および a_l を導入してネットワーク変数を分割する。
• 訓練を制約付き問題として定式化し、W_l、a_l、z_l の閉形式のサブ問題を含む Bregman/ADMM風の反復を適用する。
• W_l の更新を単純な線形最小二乗法として解く: W_l <- z_l a_l^T (a_l a_l^T)^{-1} (疑似逆数を介して)。
• Solve a_l updates with a_l = (β_{l+1} W_{l+1}^T W_{l+1} + γ_l I)^{-1} (β_{l+1} W_{l+1}^T z_{l+1} + γ_l h_l(z_l)).
• Solve z_l updates from decoupled 1D problems: minimize γ_l ||a_l − h_l(z_l)||^2 + β_l ||z_l − W_l a_{l-1}||^2, which yields closed-form or lookup solutions for piecewise linear activations (e.g., ReLU).
• Provide a Lagrange multiplier update λ <- λ + β_L (z_L − W_L a_{L-1}); discuss interpretation via Bregman iteration and the method of multipliers.

実験結果

リサーチクエスチョン

• RQ1 Can training neural networks be effectively performed without gradient-based steps?
• RQ2 Does the ADMM/Bregman-based approach scale linearly when data and computation are distributed across many cores?
• RQ3 How does the proposed method compare to SGD, CG, and L-BFGS on large-scale datasets in terms of speed and accuracy?
• RQ4 Can the framework be extended to recurrent or convolutional architectures with efficient sub-problem solutions?

主な発見

• The method decomposes training into sub-problems with closed-form solutions, avoiding gradient steps.
• Activations and weights updates decompose across layers, enabling parallelization over layers and data.
• Empirical results show linear scaling in core count, with ADMM outperforming traditional methods on very large datasets for time-to-accuracy benchmarks.
• On SVHN, ADMM achieved competitive time-to-accuracy with strong scaling up to thousands of cores compared to GPU-based methods.
• On the Higgs dataset, ADMM reached 64% accuracy significantly faster with increasing core counts (e.g., 7.8 seconds on 7200 cores) while gradient methods lagged.
• L-BFGS achieved higher final accuracy on Higgs but required far more time than ADMM for the same threshold.