[Paper Review] An Efficient Proximal-Gradient Method for Single and Multi-task Regression with Structured Sparsity

Xi Chen, Qihang Lin+4 more|arXiv (Cornell University)|Jun 30, 2018

Sparse and Compressive Sensing Techniques17 references20 citations

TL;DR

This paper proposes an efficient proximal-gradient method for single and multi-task regression with overlapping group-structured sparsity, leveraging a smooth approximation of the non-smooth structured-sparsity-inducing norm to enable faster convergence and superior scalability compared to SOCP-based approaches. The method achieves state-of-the-art performance on large-scale genetic association datasets.

ABSTRACT

We consider the optimization problem of learning regression models with a mixed-norm penalty that is defined over overlapping groups to achieve structured sparsity. It has been previously shown that such penalty can encode prior knowledge on the input or output structure to learn an structuredsparsity pattern in the regression parameters. However, because of the non-separability of the parameters of the overlapping groups, developing an efficient optimization method has remained a challenge. An existing method casts this problem as a second-order cone programming (SOCP) and solves it by interior-point methods. However, this approach is computationally expensive even for problems of moderate size. In this paper, we propose an efficient proximal-gradientmethod that achieves a faster convergence rate and is much more efficient and scalable than solving the SOCP formulation. Our method exploits the structure of the non-smooth structured-sparsity-inducing norm, introduces its smooth approximation, and solves this approximation function instead of optimizing the original objective function directly. We demonstrate the efficiency and scalability of our method on simulated datasets and show that our method can be successfully applied to a very large-scale dataset in genetic association analysis.

Motivation & Objective

• To address the computational inefficiency of existing SOCP-based methods for optimizing mixed-norm penalties with overlapping groups.
• To develop a scalable and fast optimization method for structured sparsity in single and multi-task regression.
• To exploit the structure of non-smooth, overlapping group norms to enable efficient optimization.
• To demonstrate the method's effectiveness on large-scale real-world datasets, particularly in genetic association analysis.

Proposed method

• The method introduces a smooth approximation of the non-smooth structured-sparsity-inducing norm to enable efficient optimization.
• It formulates the optimization problem using a proximal-gradient framework that handles the non-smoothness via iterative updates.
• The approach leverages the overlapping group structure to design a computationally efficient proximal operator.
• The algorithm is designed to converge faster than interior-point methods used in SOCP formulations.
• It avoids the high computational cost of SOCP by directly optimizing a smooth approximation of the original objective.
• The method is scalable and applicable to very large-scale datasets, such as those in genetic association studies.

Experimental results

Research questions

• RQ1 Can a proximal-gradient method with smooth approximation outperform SOCP-based methods in terms of convergence speed and scalability for structured sparsity problems?
• RQ2 How effective is the smooth approximation of the overlapping group norm in preserving structured sparsity patterns?
• RQ3 Can the proposed method scale efficiently to large-scale genetic association datasets with high-dimensional features?
• RQ4 Does the method maintain or improve prediction accuracy compared to existing SOCP-based approaches?

Key findings

• The proposed proximal-gradient method achieves significantly faster convergence than SOCP-based interior-point methods.
• The method demonstrates superior scalability and is capable of handling very large-scale datasets, such as those in genetic association analysis.
• The smooth approximation of the structured-sparsity-inducing norm enables efficient optimization without sacrificing sparsity structure.
• Empirical results on simulated datasets confirm the method's ability to recover structured sparsity patterns effectively.