[Paper Review] Training Neural Networks Without Gradients: A Scalable ADMM Approach

Gavin Taylor, Ryan Burmeister+4 more|arXiv (Cornell University)|May 6, 2016

Stochastic Gradient Optimization Techniques26 references146 citations

TL;DR

Introduces an ADMM/Bregman-based method to train neural networks without gradient descent, enabling linear scaling across thousands of cores and robust performance on large datasets.

ABSTRACT

With the growing importance of large network models and enormous training datasets, GPUs have become increasingly necessary to train neural networks. This is largely because conventional optimization algorithms rely on stochastic gradient methods that don't scale well to large numbers of cores in a cluster setting. Furthermore, the convergence of all gradient methods, including batch methods, suffers from common problems like saturation effects, poor conditioning, and saddle points. This paper explores an unconventional training method that uses alternating direction methods and Bregman iteration to train networks without gradient descent steps. The proposed method reduces the network training problem to a sequence of minimization sub-steps that can each be solved globally in closed form. The proposed method is advantageous because it avoids many of the caveats that make gradient methods slow on highly non-convex problems. The method exhibits strong scaling in the distributed setting, yielding linear speedups even when split over thousands of cores.

Motivation & Objective

• Motivate and address limitations of gradient-based training for large-scale neural networks.
• Propose an alternating minimization framework that decomposes training into solvable sub-steps with closed-form solutions.
• Demonstrate scalability in distributed settings and compare performance to standard gradient-based methods on large datasets.
• Provide practical guidance on implementation, initialization, and parameter choices.
• Discuss theoretical interpretations and potential extensions to recurrent and convolutional networks.

Proposed method

• Split the network variables by introducing auxiliary variables z_l and a_l to decouple W_l from activations.
• Formulate training as a constrained problem and apply a Bregman/ADMM-like iteration with closed-form sub-problems for W_l, a_l, and z_l.
• Solve W_l updates as simple linear least-squares: W_l <- z_l a_l^T (a_l a_l^T)^{-1} (via pseudoinverse).
• Solve a_l updates with a_l = (β_{l+1} W_{l+1}^T W_{l+1} + γ_l I)^{-1} (β_{l+1} W_{l+1}^T z_{l+1} + γ_l h_l(z_l)).
• Solve z_l updates from decoupled 1D problems: minimize γ_l ||a_l − h_l(z_l)||^2 + β_l ||z_l − W_l a_{l-1}||^2, which yields closed-form or lookup solutions for piecewise linear activations (e.g., ReLU).
• Provide a Lagrange multiplier update λ <- λ + β_L (z_L − W_L a_{L-1}); discuss interpretation via Bregman iteration and the method of multipliers.

Experimental results

Research questions

• RQ1 Can training neural networks be effectively performed without gradient-based steps?
• RQ2 Does the ADMM/Bregman-based approach scale linearly when data and computation are distributed across many cores?
• RQ3 How does the proposed method compare to SGD, CG, and L-BFGS on large-scale datasets in terms of speed and accuracy?
• RQ4 Can the framework be extended to recurrent or convolutional architectures with efficient sub-problem solutions?

Key findings

• The method decomposes training into sub-problems with closed-form solutions, avoiding gradient steps.
• Activations and weights updates decompose across layers, enabling parallelization over layers and data.
• Empirical results show linear scaling in core count, with ADMM outperforming traditional methods on very large datasets for time-to-accuracy benchmarks.
• On SVHN, ADMM achieved competitive time-to-accuracy with strong scaling up to thousands of cores compared to GPU-based methods.
• On the Higgs dataset, ADMM reached 64% accuracy significantly faster with increasing core counts (e.g., 7.8 seconds on 7200 cores) while gradient methods lagged.
• L-BFGS achieved higher final accuracy on Higgs but required far more time than ADMM for the same threshold.