[Paper Review] Smoothing Proximal Gradient Method for General Structured Sparse Learning

Xi Chen, Qihang Lin+3 more|arXiv (Cornell University)|Feb 14, 2012

Sparse and Compressive Sensing Techniques23 references80 citations

TL;DR

This paper proposes a smoothing proximal gradient method to efficiently solve structured sparse learning problems with non-separable penalties such as overlapping group lasso and graph-guided fusion. By leveraging Nesterov's smoothing technique, the method achieves faster convergence than subgradient methods and better scalability than interior-point methods, demonstrating strong performance on high-dimensional regression tasks with structured sparsity.

ABSTRACT

We study the problem of learning high dimensional regression models regularized by a structured-sparsity-inducing penalty that encodes prior structural information on either input or output sides. We consider two widely adopted types of such penalties as our motivating examples: 1) overlapping group lasso penalty, based on the l1/l2 mixed-norm penalty, and 2) graph-guided fusion penalty. For both types of penalties, due to their non-separability, developing an efficient optimization method has remained a challenging problem. In this paper, we propose a general optimization approach, called smoothing proximal gradient method, which can solve the structured sparse regression problems with a smooth convex loss and a wide spectrum of structured-sparsity-inducing penalties. Our approach is based on a general smoothing technique of Nesterov. It achieves a convergence rate faster than the standard first-order method, subgradient method, and is much more scalable than the most widely used interior-point method. Numerical results are reported to demonstrate the efficiency and scalability of the proposed method.

Motivation & Objective

• To address the challenge of optimizing high-dimensional regression models regularized by non-separable structured-sparsity-inducing penalties.
• To develop a scalable and efficient optimization framework applicable to a wide range of structured sparsity penalties.
• To overcome the limitations of subgradient methods (slow convergence) and interior-point methods (poor scalability) in structured sparse learning.
• To enable effective learning when prior structural information is available on input or output variables.
• To provide a general-purpose optimization method that works with smooth convex loss functions and diverse structured penalties.

Proposed method

• The method employs Nesterov's smoothing technique to approximate non-smooth structured penalties, enabling gradient-based optimization.
• It combines smoothing with proximal gradient descent to handle the non-smooth components of the objective function.
• The approach transforms the original non-smooth problem into a sequence of smooth subproblems that are solved efficiently.
• A line search strategy is used to adaptively adjust step sizes, ensuring convergence and stability.
• The algorithm is designed to be applicable to both overlapping group lasso and graph-guided fusion penalties.
• The method achieves an optimal convergence rate of O(1/k²) for smooth convex problems, outperforming standard subgradient methods.

Experimental results

Research questions

• RQ1 Can a general optimization framework be developed for structured sparse learning with non-separable penalties?
• RQ2 How can the convergence speed of first-order methods be improved for non-smooth structured sparsity problems?
• RQ3 Can the proposed method achieve better scalability than interior-point methods while maintaining high accuracy?
• RQ4 Does the smoothing proximal gradient method outperform standard subgradient methods in terms of convergence rate and computational efficiency?
• RQ5 To what extent can the method be generalized across different types of structured-sparsity-inducing penalties?

Key findings

• The smoothing proximal gradient method achieves a convergence rate of O(1/k²), which is faster than the O(1/√k) rate of standard subgradient methods.
• The method demonstrates superior scalability compared to interior-point methods, enabling efficient solution of large-scale structured sparse learning problems.
• Numerical experiments show that the method converges significantly faster than subgradient-based approaches on both overlapping group lasso and graph-guided fusion problems.
• The algorithm effectively handles non-separable penalties such as l1/l2 mixed-norm and graph-structured fusion, maintaining high accuracy.
• The approach is robust across different problem sizes and achieves consistent performance on high-dimensional datasets.
• The method is general enough to be applied to a wide spectrum of structured-sparsity-inducing penalties beyond the two motivating examples.