[Paper Review] Convex and Network Flow Optimization for Structured Sparsity

Julien Mairal, Rodolphe Jenatton+2 more|arXiv (Cornell University)|Apr 11, 2011

Sparse and Compressive Sensing Techniques86 references83 citations

TL;DR

This paper proposes efficient convex optimization methods for structured sparsity with overlapping groups, using network flow techniques to compute proximal operators exactly and accelerating convergence via FISTA. It achieves significant speedups over existing methods by reducing the problem to quadratic min-cost flow, enabling scalable solutions for tasks like matrix factorization, dictionary learning, and image denoising.

ABSTRACT

We consider a class of learning problems regularized by a structured sparsity-inducing norm defined as the sum of l_2- or l_infinity-norms over groups of variables. Whereas much effort has been put in developing fast optimization techniques when the groups are disjoint or embedded in a hierarchy, we address here the case of general overlapping groups. To this end, we present two different strategies: On the one hand, we show that the proximal operator associated with a sum of l_infinity-norms can be computed exactly in polynomial time by solving a quadratic min-cost flow problem, allowing the use of accelerated proximal gradient methods. On the other hand, we use proximal splitting techniques, and address an equivalent formulation with non-overlapping groups, but in higher dimension and with additional constraints. We propose efficient and scalable algorithms exploiting these two strategies, which are significantly faster than alternative approaches. We illustrate these methods with several problems such as CUR matrix factorization, multi-task learning of tree-structured dictionaries, background subtraction in video sequences, image denoising with wavelets, and topographic dictionary learning of natural image patches.

Motivation & Objective

• To address the challenge of optimizing structured sparsity regularizations with general overlapping groups, which are difficult to handle with standard convex optimization techniques.
• To develop a fast and exact method for computing the proximal operator of a sum of ℓ∞-norms over overlapping groups.
• To enable scalable and efficient optimization for large-scale machine learning problems involving structured sparsity, such as multi-task learning and dictionary learning.
• To establish a connection between structured sparsity and network flow optimization, leveraging min-cost flow algorithms for exact computation.
• To provide practical algorithms that outperform existing approaches in speed and scalability across diverse applications.

Proposed method

• The proximal operator for the sum of ℓ∞-norms over overlapping groups is computed by solving a quadratic min-cost flow problem, enabling exact and efficient updates in proximal gradient methods.
• A duality gap computation is introduced using the dual norm of the regularization, allowing for accurate convergence monitoring and stopping criteria.
• An alternative formulation with non-overlapping groups in a higher-dimensional space is used, combined with proximal splitting techniques and the alternating direction method of multipliers (ADMM).
• The FISTA algorithm is employed with line search and adaptive Lipschitz constant estimation to accelerate convergence.
• The network flow formulation is linked to parametric max-flow problems, enabling comparison with state-of-the-art max-flow solvers like GGT and SIMP.
• The approach is implemented and benchmarked on real-world problems, including CUR matrix factorization, video background subtraction, and wavelet-based image denoising.

Experimental results

Research questions

• RQ1 Can the proximal operator for a sum of ℓ∞-norms over overlapping groups be computed exactly and efficiently?
• RQ2 How can network flow optimization be leveraged to accelerate proximal methods in structured sparsity problems?
• RQ3 What is the performance gain of using min-cost flow-based proximal computation compared to existing solvers for structured sparsity?
• RQ4 Can the proposed methods scale effectively to large-scale problems such as high-dimensional matrix factorization and image processing tasks?
• RQ5 How does the duality gap computation based on the dual norm improve convergence monitoring in these optimization problems?

Key findings

• The proximal operator for overlapping group sparsity with ℓ∞-norms is computed exactly by solving a quadratic min-cost flow problem, enabling efficient proximal gradient methods.
• The proposed ProxFlow method achieves average execution times of 0.4s, 3.1s, and 113.0s for problems with 10⁴, 10⁵, and 10⁶ variables, respectively, outperforming GGT and SIMP across all scales.
• For a 57,600-pixel video background subtraction task, ProxFlow runs in 1.7s, compared to 8.31s (SIMP) and 16.7s (GGT).
• The duality gap computation, based on the dual norm evaluation, enables accurate convergence monitoring and is efficiently computed via the same network flow framework.
• The method scales effectively to large problems, with consistent speed advantages across different regularization regimes and problem sizes.
• The approach enables practical application of structured sparsity in tasks such as CUR matrix factorization, hierarchical dictionary learning, and wavelet-based image denoising, with demonstrated speed and accuracy improvements.