[Paper Review] DINGO: Distributed Newton-Type Method for Gradient-Norm Optimization

Rixon Crane, Fred Roosta|arXiv (Cornell University)|Jan 16, 2019

Sparse and Compressive Sensing TechniquesEngineering27 references20 citations

TL;DR

DINGO is a communication-efficient, distributed Newton-type optimization algorithm that minimizes the norm of the gradient by solving simple linear least-squares subproblems. It achieves global convergence with guaranteed reduction in gradient norm regardless of hyper-parameter choice, outperforming first-order and other second-order methods in communication efficiency and stability across diverse, non-convex problems with arbitrary data distribution.

ABSTRACT

For optimization of a sum of functions in a distributed computing environment, we present a novel communication efficient Newton-type algorithm that enjoys a variety of advantages over similar existing methods. Similar to Newton-MR, our algorithm, DINGO, is derived by optimization of the gradient's norm as a surrogate function. DINGO does not impose any specific form on the underlying functions, and its application range extends far beyond convexity. In addition, the distribution of the data across the computing environment can be arbitrary. Further, the underlying sub-problems of DINGO are simple linear least-squares, for which a plethora of efficient algorithms exist. Lastly, DINGO involves a few hyper-parameters that are easy to tune. Moreover, we theoretically show that DINGO is not sensitive to the choice of its hyper-parameters in that a strict reduction in the gradient norm is guaranteed, regardless of the selected hyper-parameters. We demonstrate empirical evidence of the effectiveness, stability and versatility of our method compared to other relevant algorithms.

Motivation & Objective

• To develop a distributed optimization algorithm that reduces communication overhead in large-scale machine learning and scientific computing.
• To extend second-order optimization to non-convex and non-structured problems without requiring specific functional forms.
• To ensure robust convergence with minimal hyper-parameter sensitivity in distributed environments.
• To enable efficient use of distributed computational resources by minimizing communication rounds.
• To provide a practical, implementable method based on simple linear least-squares subproblems.

Proposed method

• DINGO formulates optimization as minimizing the norm of the gradient, using a surrogate objective to guide search direction.
• It employs a distributed inexact Newton method where each iteration solves a linear least-squares problem to compute the search direction.
• The algorithm uses backtracking line search for step size selection, ensuring sufficient decrease in the gradient norm.
• Communication is minimized by exchanging only gradient and Hessian-related information between worker nodes and a central driver.
• The method is designed to be robust: a strict reduction in the gradient norm is guaranteed regardless of hyper-parameter values.
• It supports arbitrary data partitioning across workers and does not require convexity or smoothness beyond standard differentiability.

Experimental results

Research questions

• RQ1 Can a distributed second-order method be designed that is both communication-efficient and applicable beyond convex problems?
• RQ2 How can the Hessian approximation be computed efficiently in a distributed setting with minimal communication?
• RQ3 Does a Newton-type method based on gradient-norm minimization ensure global convergence with minimal hyper-parameter sensitivity?
• RQ4 How does DINGO compare to first-order and other second-order methods in terms of communication efficiency and convergence speed?
• RQ5 Can DINGO effectively scale across heterogeneous and distributed computing environments with varying network latencies?

Key findings

• DINGO completed 12 iterations per hour on AWS, a 50% increase from 8 iterations on a single node, demonstrating effective scaling in distributed environments.
• In comparison to Sync-SGD, DINGO reduced communication costs by over 80% on AWS, completing significantly fewer iterations but with faster convergence in gradient norm.
• DINGO achieved faster reduction in gradient norm than GIANT, DiSCO, InexactDANE, and AIDE, even with fewer communication rounds.
• Empirical results show DINGO maintains stable and fast convergence across multiple datasets (CIFAR10, EMNIST) and problem types (softmax regression).
• The algorithm's performance is robust to hyper-parameter choice: different values of the line search parameter θ yielded similar objective convergence, but θ=10⁻⁴ gave the fastest gradient norm reduction.
• DINGO outperformed all baselines in reducing the gradient norm, even in non-convex settings, with guaranteed convergence independent of hyper-parameter tuning.