[Paper Review] On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyond

Xiao–Tong Yuan, Ping Li|arXiv (Cornell University)|Aug 6, 2019

Advanced Optimization Algorithms Research14 citations

TL;DR

This paper proposes enhanced variants of the DANE algorithm—DANE with backtracking line search and DANE-HB (Heavy-Ball)—to achieve global convergence and sharper local convergence rates for both quadratic and non-quadratic strongly convex functions. It establishes nearly tight local convergence rates and global convergence for linear models, with improved communication and computational complexity bounds via Lyapunov analysis.

ABSTRACT

The DANE algorithm is an approximate Newton method popularly used for communication-efficient distributed machine learning. Reasons for the interest in DANE include scalability and versatility. Convergence of DANE, however, can be tricky; its appealing convergence rate is only rigorous for quadratic objective, and for more general convex functions the known results are no stronger than those of the classic first-order methods. To remedy these drawbacks, we propose in this paper some new alternatives of DANE which are more suitable for analysis. We first introduce a simple variant of DANE equipped with backtracking line search, for which global asymptotic convergence and sharper local non-asymptotic convergence rate guarantees can be proved for both quadratic and non-quadratic strongly convex functions. Then we propose a heavy-ball method to accelerate the convergence of DANE, showing that nearly tight local rate of convergence can be established for strongly convex functions, and with proper modification of algorithm the same result applies globally to linear prediction models. Numerical evidence is provided to confirm the theoretical and practical advantages of our methods.

Motivation & Objective

• To address the lack of global convergence and weak local convergence guarantees in standard DANE for non-quadratic objectives.
• To develop new DANE variants with provable global asymptotic convergence and tighter local non-asymptotic convergence rates.
• To extend convergence guarantees to linear prediction models using Heavy-Ball acceleration.
• To provide a unified Lyapunov-based framework for analyzing convergence in distributed approximate Newton methods.
• To improve communication and computational complexity bounds through algorithmic enhancements.

Proposed method

• Introduces a backtracking line search variant of DANE to ensure global convergence and improve local convergence rates.
• Proposes DANE-HB, a Heavy-Ball-accelerated DANE variant that achieves nearly tight local convergence rates for strongly convex functions.
• Uses a Lyapunov function framework to prove convergence, enabling sharper bounds and unified analysis.
• Employs SVRG as the local solver for subproblems, with IFO complexity used as the computational metric.
• Derives communication complexity bounds of order $\mathcal{O}\left(\frac{\sqrt{\kappa}}{n^{1/4}}\log\left(\frac{1}{\epsilon}\right)\right)$ for quadratic objectives.
• Modifies DANE-HB to achieve global convergence for linear models by adjusting the algorithmic structure.

Experimental results

Research questions

• RQ1 Can DANE be modified to achieve global convergence for non-quadratic strongly convex functions?
• RQ2 What is the sharpest possible local non-asymptotic convergence rate achievable by approximate Newton methods in distributed settings?
• RQ3 Can Heavy-Ball acceleration be effectively integrated into DANE to improve convergence speed?
• RQ4 How do the communication and computational complexities of the new DANE variants compare to first-order methods like SVRG?
• RQ5 Can the proposed methods maintain convergence guarantees across different model classes, including linear models?

Key findings

• The proposed backtracking DANE variant achieves global asymptotic convergence and sharper local non-asymptotic convergence rates for both quadratic and non-quadratic strongly convex functions.
• DANE-HB achieves nearly tight local convergence rates for strongly convex functions, with theoretical guarantees matching known lower bounds.
• With proper modification, DANE-HB ensures global convergence for linear prediction models, extending applicability beyond quadratic cases.
• The communication complexity of DANE-HB is $\mathcal{O}\left(\frac{\sqrt{\kappa}}{n^{1/4}}\log\left(\frac{1}{\epsilon}\right)\right)$, improving upon standard DANE and matching nearly tight bounds.
• The IFO complexity of DANE-HB is $\mathcal{O}\left(\sqrt{\kappa}\left(n^{3/4}+n^{1/4}\right)\log^{2}\left(\frac{1}{\epsilon}\right)\right)$, offering up to $m^{3/4}$ computational savings over SVRG in large-scale settings.
• Numerical results confirm the theoretical advantages, showing faster convergence and improved communication efficiency in practice.