[Paper Review] On the non-ergodic convergence rate of an inexact augmented Lagrangian framework for composite convex programming

Ya‐Feng Liu, Xin Liu+1 more|arXiv (Cornell University)|Mar 18, 2016

Sparse and Compressive Sensing Techniques18 references3 citations

TL;DR

This paper proposes an inexact augmented Lagrangian (IAL) framework for composite convex optimization with linear constraints, where the augmented Lagrangian subproblems are solved approximately using a weaker stopping criterion. It establishes global convergence and non-ergodic convergence rate guarantees, demonstrating faster CPU performance and superior sparsity in solutions compared to exact methods.

ABSTRACT

In this paper, we consider the linearly constrained composite convex optimization problem, whose objective is a sum of a smooth function and a possibly nonsmooth function. We propose an inexact augmented Lagrangian (IAL) framework for solving the problem. The stopping criterion used in solving the augmented Lagrangian (AL) subproblem in the proposed IAL framework is weaker and potentially much easier to check than the one used in most of the existing IAL frameworks/methods. We analyze the global convergence and the non-ergodic convergence rate of the proposed IAL framework.

Motivation & Objective

• To develop an inexact augmented Lagrangian (IAL) framework that allows weaker stopping criteria for solving augmented Lagrangian subproblems in composite convex optimization.
• To establish global convergence and non-ergodic convergence rate convergence for the proposed IAL framework under inexact subproblem solutions.
• To demonstrate the practical efficiency and sparsity-preserving properties of the non-ergodic solution path in comparison to ergodic counterparts.
• To justify the importance of non-ergodic convergence rate analysis through numerical results on basis pursuit problems.

Proposed method

• The IAL framework solves the augmented Lagrangian subproblem approximately, using a stopping criterion based on suboptimality gap: $\mathcal{L}_{\beta}(x^{k+1};\lambda^k) - \mathcal{L}_{\beta}(x(\lambda^k);\lambda^k) \leq \eta_k^2$, which is weaker than standard criteria.
• The method updates dual variables via $\lambda^{k+1} = \lambda^k + \beta(Ax(\lambda^k) - b)$, mimicking dual gradient ascent on the augmented Lagrangian dual function.
• Convergence analysis is conducted under the assumption that the sequence $\{\eta_k\}$ is summable, ensuring global convergence of the iterates.
• Non-ergodic convergence rates are derived for both the primal variable and the dual infeasibility, providing non-averaged convergence guarantees.
• The framework is applied to basis pursuit problems, where the nonsmooth term $g(x)$ is the indicator function of a convex set, modeling sparse recovery.
• Numerical experiments compare the IAL framework with exact augmented Lagrangian (EAL) methods, measuring sparsity, objective error, residual, and CPU time.

Experimental results

Research questions

• RQ1 Can an inexact augmented Lagrangian framework with a weaker stopping criterion still guarantee global convergence and non-ergodic convergence rates?
• RQ2 How does the non-ergodic solution path of the IAL framework compare to the ergodic path in terms of sparsity and convergence speed?
• RQ3 What is the impact of the inexactness tolerance $\eta_k$ on the overall convergence behavior and computational efficiency?
• RQ4 Does the proposed IAL framework achieve faster CPU time than exact methods while maintaining comparable accuracy and sparsity?

Key findings

• The IAL framework achieves global convergence under a summable sequence of inexactness tolerances $\eta_k$, even when subproblems are solved approximately.
• The non-ergodic convergence rate for the primal variable and dual infeasibility is established, providing non-averaged convergence guarantees.
• Numerical results show that IAL is significantly faster than EAL in CPU time, with IAL achieving the same accuracy in less than half the time on average.
• The non-ergodic solution $\bar{x}$ is substantially more sparse than the ergodic solution $x_e$, and its sparsity pattern perfectly matches the true solution $x^*$ in all tested instances.
• For larger problems ($m=1800, n=3000$), IAL achieves a 25-30% reduction in CPU time compared to EAL, while maintaining similar objective error and residual.
• Even when $\eta_k < 10^{-4}$ for $k > 100$, IAL remains faster than EAL, indicating that the weaker stopping criterion does not compromise efficiency.