[Paper Review] An efficient linearly convergent semismooth Netwon-CG augmented Lagrangian method for Lasso problems

Xudong Li, Defeng Sun+1 more|arXiv (Cornell University)|Jul 19, 2016

Sparse and Compressive Sensing Techniques44 references3 citations

TL;DR

This paper proposes Ssnal, a novel algorithm that combines the semismooth Newton-CG method with the augmented Lagrangian framework to solve large-scale Lasso problems efficiently. By exploiting the piecewise linear-quadratic structure of Lasso problems, Ssnal achieves global convergence and local linear convergence, outperforming state-of-the-art solvers on real-world datasets in terms of speed and robustness.

ABSTRACT

We develop a fast and robust algorithm for solving large-scale convex composite optimization models with an emphasis on the $\ell_1$-regularized least square regression (the Lasso) problems. Although there exist a large amount of solvers in the literature for Lasso problems, so far no solver can handle difficult real large scale regression problems. By relying on the piecewise linear-quadratic structure of the problems to realize the remarkable fast linear convergence property of the augmented Lagrangian algorithm, and by exploiting the superlinear convergence of the semismooth Newton-CG method, we are able to design a new algorithm, called {\sc Ssnal}, to efficiently solve the aforementioned difficult problems. Global convergence and local linear convergence results for {\sc Ssnal} are established. Numerical results, including the comparison between our approach and several state-of-the-art solvers, on real data sets, are presented to demonstrate the high efficiency and robustness of our proposed algorithm in solving large-scale difficult problems.

Motivation & Objective

• To address the challenge of solving large-scale, difficult Lasso problems that existing solvers fail to handle effectively.
• To develop an algorithm that leverages the structural properties of Lasso problems for faster convergence.
• To achieve both global convergence and locally linear convergence rates in solving Lasso problems.
• To outperform existing state-of-the-art solvers on real-world large-scale regression datasets.

Proposed method

• The algorithm uses the augmented Lagrangian method to decompose the Lasso problem into subproblems with favorable structure.
• It applies the semismooth Newton-CG method to solve the subproblems, exploiting their piecewise linear-quadratic nature for superlinear convergence.
• The method integrates the fast local convergence of semismooth Newton-CG with the global convergence guarantees of the augmented Lagrangian framework.
• The algorithm is designed to handle large-scale problems by efficiently managing the computational cost of solving the subproblems.
• The piecewise linear-quadratic structure of the Lasso problem is systematically exploited to enable fast convergence.
• Global and local convergence properties are rigorously established through theoretical analysis.

Experimental results

Research questions

• RQ1 Can a hybrid algorithm combining semismooth Newton-CG and augmented Lagrangian methods achieve faster convergence for large-scale Lasso problems?
• RQ2 How does the piecewise linear-quadratic structure of Lasso problems influence the convergence behavior of the proposed algorithm?
• RQ3 Can the proposed method outperform existing state-of-the-art solvers on real-world large-scale regression datasets?
• RQ4 What theoretical guarantees (global and local convergence) can be established for the new algorithm?
• RQ5 How does the algorithm maintain robustness and efficiency across diverse large-scale problem instances?

Key findings

• Ssnal achieves global convergence and locally linear convergence for solving Lasso problems.
• The algorithm demonstrates superior performance on real-world large-scale regression datasets compared to state-of-the-art solvers.
• The piecewise linear-quadratic structure of the Lasso problem enables the remarkable fast linear convergence of the augmented Lagrangian method.
• The integration of semismooth Newton-CG with the augmented Lagrangian framework leads to superlinear convergence in subproblem solutions.
• Numerical results confirm the high efficiency and robustness of Ssnal in solving difficult large-scale Lasso problems.
• The method effectively handles large-scale instances where existing solvers fail, demonstrating practical scalability.