[Paper Review] GIANT: Globally Improved Approximate Newton Method for Distributed Optimization

Shusen Wang, Fred Roosta+2 more|arXiv (Cornell University)|Sep 11, 2017

Stochastic Gradient Optimization Techniques75 citations

TL;DR

GIANT is a distributed Newton-type optimization method that uses locally computed approximate Newton directions averaged across workers to form a global direction, achieving communication-efficient, provably faster convergence than several first- and second-order baselines, with only one tuning parameter.

ABSTRACT

For distributed computing environment, we consider the empirical risk minimization problem and propose a distributed and communication-efficient Newton-type optimization method. At every iteration, each worker locally finds an Approximate NewTon (ANT) direction, which is sent to the main driver. The main driver, then, averages all the ANT directions received from workers to form a {\\it Globally Improved ANT} (GIANT) direction. GIANT is highly communication efficient and naturally exploits the trade-offs between local computations and global communications in that more local computations result in fewer overall rounds of communications. Theoretically, we show that GIANT enjoys an improved convergence rate as compared with first-order methods and existing distributed Newton-type methods. Further, and in sharp contrast with many existing distributed Newton-type methods, as well as popular first-order methods, a highly advantageous practical feature of GIANT is that it only involves one tuning parameter. We conduct large-scale experiments on a computer cluster and, empirically, demonstrate the superior performance of GIANT.

Motivation & Objective

• Address the computational and communication bottlenecks of distributed empirical risk minimization.
• Develop a Newton-type method that minimizes inter-node communication while leveraging local curvature information.
• Provide theoretical guarantees showing improved convergence rates compared with first-order and existing distributed Newton methods.
• Demonstrate practical performance gains on large-scale distributed datasets.

Proposed method

• Each worker computes a local approximate Newton (ANT) direction using its data subset.
• Local ANT directions are obtained via Hessian-vector products solved with conjugate gradients, avoiding explicit Hessian formation.
• The GIANT direction is the average (harmonic mean in the Hessian sense) of local ANT directions, yielding a globally improved update: p_t ≈ (1/m) sum_i H̃_{t,i}^{-1} g_t.
• Communication per iteration is limited to sending d-dimensional vectors, not d×d matrices.
• The method uses a single tuning parameter: the maximum number of CG iterations for the local solves.
• Convergence analysis covers quadratic losses with global convergence, and general smooth losses with linear-quadratic local convergence, under standard Lipschitz Hessian assumptions.

Experimental results

Research questions

• RQ1 Can GIANT achieve global convergence for quadratic objectives and improved convergence rates in distributed settings compared to existing second-order methods?
• RQ2 How does the harmonic-mean Hessian approximation affect communication complexity and practical performance when aggregating locally computed directions?
• RQ3 What are the convergence guarantees when local subproblem solutions are solved inexactly (e.g., via CG) and how do they compare to exact solutions?
• RQ4 How does GIANT perform empirically against established baselines (AGD, L-BFGS, DANE) on large-scale, real-world datasets?

Key findings

• GIANT achieves a communication-efficient update where per-iteration communication scales with d rather than d^2, via averaging local directions and avoiding explicit Hessian transmission.
• For quadratic losses, GIANT attains global convergence with a logarithmic dependence on the condition number, improving over prior distributed Newton methods.
• For general smooth losses, GIANT exhibits linear-quadratic local convergence, with the linear term driven by Hessian approximation and the quadratic term by non-quadratic objective effects.
• GIANT demonstrates superior empirical performance on large-scale logistic regression tasks across multiple datasets, outperforming AGD, L-BFGS, and DANE in training objective value and test error within the same wall-clock time.
• The method requires only one tuning parameter (max CG iterations) and supports inexact local solves without sacrificing convergence guarantees.
• Line search added in experiments maintains robustness and does not require extra tuning, preserving the overall simplicity of GIANT.