[Paper Review] Curvature-aided Incremental Aggregated Gradient Method

Hoi-To Wai, Wei Shi+2 more|arXiv (Cornell University)|Oct 24, 2017

Sparse and Compressive Sensing Techniques12 references3 citations

TL;DR

This paper proposes the curvature-aided incremental aggregated gradient (CIAG) method, a first-order optimization algorithm that leverages incremental Hessian information to accelerate convergence in finite sum problems. It achieves a linear convergence rate comparable to full gradient methods with only ${\cal O}(d^2)$ per-iteration complexity, outperforming IAG in iteration count and IN in computational efficiency for high-dimensional problems.

ABSTRACT

We propose a new algorithm for finite sum optimization which we call the curvature-aided incremental aggregated gradient (CIAG) method. Motivated by the problem of training a classifier for a d-dimensional problem, where the number of training data is $m$ and $m \gg d \gg 1$, the CIAG method seeks to accelerate incremental aggregated gradient (IAG) methods using aids from the curvature (or Hessian) information, while avoiding the evaluation of matrix inverses required by the incremental Newton (IN) method. Specifically, our idea is to exploit the incrementally aggregated Hessian matrix to trace the full gradient vector at every incremental step, therefore achieving an improved linear convergence rate over the state-of-the-art IAG methods. For strongly convex problems, the fast linear convergence rate requires the objective function to be close to quadratic, or the initial point to be close to optimal solution. Importantly, we show that running one iteration of the CIAG method yields the same improvement to the optimality gap as running one iteration of the full gradient method, while the complexity is $O(d^2)$ for CIAG and $O(md)$ for the full gradient. Overall, the CIAG method strikes a balance between the high computation complexity incremental Newton-type methods and the slow IAG method. Our numerical results support the theoretical findings and show that the CIAG method often converges with much fewer iterations than IAG, and requires much shorter running time than IN when the problem dimension is high.

Motivation & Objective

• Address the slow convergence of incremental aggregated gradient (IAG) methods in large-scale finite sum optimization.
• Overcome the high computational cost of incremental Newton (IN) methods that require Hessian matrix inversion.
• Develop a method that combines the low per-iteration cost of IAG with the fast convergence of full gradient methods.
• Achieve linear convergence with minimal storage and computation overhead, especially for $m \gg d \gg 1$ settings.
• Ensure numerical stability and scalability in high-dimensional problems where IN fails to converge.

Proposed method

• Introduce an incremental Hessian approximation to trace the full gradient vector at each step.
• Use curvature (Hessian) information to guide gradient updates without computing matrix inverses.
• Maintain a running estimate of the aggregated Hessian over component functions to improve convergence.
• Apply a step size rule that asymptotically matches the convergence rate of full gradient iterations.
• Ensure global convergence under strong convexity with sufficiently small step sizes.
• Propose an adaptive step size strategy to achieve accelerated convergence in practice.

Experimental results

Research questions

• RQ1 Can curvature information be used to accelerate incremental gradient methods without inverting Hessian matrices?
• RQ2 Does the proposed CIAG method achieve a linear convergence rate comparable to full gradient methods?
• RQ3 How does CIAG compare in convergence speed and computational cost to IAG and IN in high-dimensional settings?
• RQ4 Can CIAG maintain numerical stability and convergence when IN fails due to ill-conditioned Hessian inversion?
• RQ5 What is the trade-off between convergence rate and per-iteration complexity in incremental optimization?

Key findings

• CIAG achieves a linear convergence rate of $1 - 4Q/(Q+1)^2$, matching the full gradient method's rate, with ${\cal O}(d^2)$ per-iteration complexity.
• For quadratic-like objectives or initial points near the optimum, one CIAG iteration improves the optimality gap as much as one full gradient iteration.
• Numerical results show CIAG converges in significantly fewer iterations than IAG and in less time than IN, especially for high-dimensional problems ($d=501$, $m=2000$).
• The IN method fails to converge in high-dimensional cases (e.g., $d=501$), while CIAG remains numerically stable and efficient.
• On real datasets (mushrooms, w8a, alpha), CIAG reaches convergence in fewer effective passes than IAG and in less time than IN, despite IN’s faster iteration count.
• For the alpha dataset ($d=500$, $m=500,000$), CIAG required 7.6 effective passes and 463.08 seconds, while IN required 2.3 passes but 1130.7 seconds due to high Hessian inversion cost.