[Paper Review] Network Flow Algorithms for Structured Sparsity

Julien Mairal, Rodolphe Jenatton+2 more|arXiv (Cornell University)|Aug 31, 2010

Sparse and Compressive Sensing Techniques54 references122 citations

TL;DR

This paper proposes a novel network flow-based algorithm to efficiently compute the proximal operator for structured sparsity-inducing norms defined as sums of ℓ∞-norms over overlapping groups. By showing that the proximal problem is dual to a quadratic min-cost flow problem, the method enables exact, polynomial-time solutions for large-scale problems with millions of variables, significantly improving scalability over prior approaches.

ABSTRACT

We consider a class of learning problems that involve a structured sparsity-inducing norm defined as the sum of $\ell_\infty$-norms over groups of variables. Whereas a lot of effort has been put in developing fast optimization methods when the groups are disjoint or embedded in a specific hierarchical structure, we address here the case of general overlapping groups. To this end, we show that the corresponding optimization problem is related to network flow optimization. More precisely, the proximal problem associated with the norm we consider is dual to a quadratic min-cost flow problem. We propose an efficient procedure which computes its solution exactly in polynomial time. Our algorithm scales up to millions of variables, and opens up a whole new range of applications for structured sparse models. We present several experiments on image and video data, demonstrating the applicability and scalability of our approach for various problems.

Motivation & Objective

• To address the lack of efficient optimization methods for structured sparsity with general overlapping groups.
• To develop a scalable and exact algorithm for computing the proximal operator of overlapping group sparsity norms.
• To establish a theoretical connection between structured sparsity and network flow optimization.
• To enable practical application of structured sparse models in high-dimensional settings such as image and video analysis.

Proposed method

• The proximal operator for the overlapping group sparsity norm is shown to be equivalent to solving a quadratic min-cost flow problem.
• The method leverages duality between the proximal problem and a network flow formulation, allowing exact solution via min-cost flow algorithms.
• A specialized network flow construction is designed to model the ℓ∞-norms over arbitrary overlapping groups.
• The algorithm is implemented using a parametric max-flow solver, with performance comparisons to existing methods.
• The dual norm is efficiently evaluated, enabling computation of duality gaps for convergence monitoring.
• The approach is integrated into FISTA for solving large-scale regularized learning problems.

Experimental results

Research questions

• RQ1 Can the proximal operator for overlapping group sparsity norms be computed efficiently and exactly in polynomial time?
• RQ2 Is there a theoretical duality between structured sparsity regularization and network flow optimization?
• RQ3 How does the proposed method scale to problems with millions of variables compared to existing approaches?
• RQ4 Can the method be effectively applied to real-world image and video learning tasks?
• RQ5 Can duality gaps be efficiently computed to monitor convergence in large-scale optimization?

Key findings

• The proposed ProxFlow algorithm computes the proximal operator in polynomial time, enabling exact solutions for large-scale problems.
• The method outperforms existing parametric max-flow solvers (GGT and SIMP) on all tested benchmarks, including datasets with up to 1 million variables.
• Execution times were 0.4s for 10,000 variables, 3.1s for 100,000, and 113.0s for 1 million variables, demonstrating strong scalability.
• The algorithm achieved faster convergence than GGT and SIMP across different regularization regimes.
• The dual norm evaluation enables accurate duality gap computation, supporting reliable convergence monitoring.
• The approach enables new applications in video background subtraction and hierarchical dictionary learning for image patches.