[Paper Review] Graph-Structured Multi-task Regression and an Efficient Optimization Method for General Fused Lasso Manuscript

Xi Chen, Se Young Kim+3 more|arXiv (Cornell University)|Jun 30, 2018

Sparse and Compressive Sensing Techniques19 references66 citations

TL;DR

This paper proposes Graph-Structured Multi-task Regression using Graph-guided Fused Lasso (GFlasso), which leverages a graph structure over output variables to encourage correlated outputs to share relevant inputs via a fusion penalty. The authors develop a fast, scalable proximal-gradient method that outperforms standard first-order and second-order methods in convergence and scalability, while maintaining theoretical consistency and superior performance on structured multi-task learning tasks.

ABSTRACT

We consider the problem of learning a structured multi-task regression, where the output consists of multiple responses that are related by a graph and the correlated response variables are dependent on the common inputs in a sparse but synergistic manner. Previous methods such as l1/l2 -regularized multi-task regression assume that all of the output variables are equally related to the inputs, although in many real-world problems, outputs are related in a complex manner. In this paper, we propose graph-guided fused lasso (GFlasso) for structured multi-task regression that exploits the graph structure over the output variables. We introduce a novel penalty function based on fusion penalty to encourage highly correlated outputs to share a common set of relevant inputs. In addition, we propose a simple yet efficient proximal-gradient method for optimizing GFlasso that can also be applied to any optimization problems with a convex smooth loss and the general class of fusion penalty defined on arbitrary graph structures. By exploiting the structure of the non-smooth “fusion penalty”, our method achieves a faster convergence rate than the standard first-order method, sub-gradient method, and is significantly more scalable than the widely adopted second-order cone-programming and quadratic-programming formulations. In addition, we provide an analysis of the consistency property of the GFlasso model. Experimental results not only demonstrate the superiority of GFlasso over the standard lasso but also show the efficiency and scalability of our proximal-gradient method.

Motivation & Objective

• Address the limitation of standard l1/l2 multi-task regression, which assumes uniform relationships between all outputs and inputs, by modeling complex, structured dependencies among outputs.
• Exploit graph-structured relationships among multiple response variables to improve learning efficiency and sparsity in input selection.
• Develop an efficient optimization algorithm tailored for general fused lasso problems on arbitrary graph structures, enabling scalable inference.
• Ensure theoretical consistency of the GFlasso model under appropriate conditions, supporting reliable estimation in high-dimensional settings.
• Demonstrate that structured fusion penalties lead to better performance than standard lasso or multi-task lasso in real-world multi-response regression tasks.

Proposed method

• Propose a novel fusion penalty based on graph structure to encourage correlated outputs to adopt similar input coefficient patterns.
• Formulate the GFlasso objective as a convex optimization problem combining a smooth loss function with a general fusion penalty over arbitrary graphs.
• Design a proximal-gradient method that exploits the structure of the non-smooth fusion penalty to accelerate convergence beyond standard sub-gradient or first-order methods.
• Enable the optimization method to be applicable to any problem with a convex smooth loss and fusion penalty on arbitrary graph topologies.
• Utilize proximal operators tailored to the fusion penalty to efficiently compute updates in each iteration, reducing computational overhead.
• Integrate the method into a first-order optimization framework with line search, ensuring convergence while maintaining scalability.

Experimental results

Research questions

• RQ1 Can incorporating graph-structured relationships among output variables improve the performance of multi-task regression models?
• RQ2 Does the proposed fusion penalty lead to better sparsity and shared input selection among correlated outputs compared to standard lasso or multi-task lasso?
• RQ3 Can a proximal-gradient method be designed to efficiently optimize general fused lasso problems on arbitrary graphs, outperforming existing second-order and first-order approaches?
• RQ4 Is the GFlasso model consistent under appropriate regularity conditions, ensuring reliable estimation as sample size increases?
• RQ5 How does the proposed method scale with increasing problem size and graph complexity compared to standard quadratic or second-order cone programming formulations?

Key findings

• The GFlasso model achieves superior performance over standard lasso and multi-task lasso in structured multi-task regression tasks, particularly when outputs exhibit correlated relationships.
• The proposed proximal-gradient method converges significantly faster than standard sub-gradient and first-order methods due to exploitation of the fusion penalty's structure.
• The method is substantially more scalable than traditional second-order cone programming and quadratic programming formulations, enabling application to large-scale problems.
• Theoretical analysis confirms the consistency of the GFlasso estimator, supporting its reliability in high-dimensional settings.
• Empirical results demonstrate that GFlasso effectively identifies shared relevant inputs among correlated outputs, improving model interpretability and predictive accuracy.
• The optimization framework is general and applicable to any fused lasso problem with a convex smooth loss and arbitrary graph structure, enhancing its practical utility.