[Paper Review] Efficient Minimization of Decomposable Submodular Functions

Peter Stobbe, Andreas Krause|arXiv (Cornell University)|Oct 26, 2010

Complexity and Algorithms in Graphs18 references91 citations

TL;DR

This paper introduces SLG, a novel algorithm for efficiently minimizing decomposable submodular functions—those expressible as sums of concave functions applied to modular functions. By leveraging smoothed convex optimization on the Lovász extension, SLG achieves orders-of-magnitude speedups over state-of-the-art general-purpose submodular minimization algorithms, scaling to problems with tens of thousands of variables, as demonstrated on synthetic benchmarks and a joint classification-and-segmentation task in image processing.

ABSTRACT

Many combinatorial problems arising in machine learning can be reduced to the problem of minimizing a submodular function. Submodular functions are a natural discrete analog of convex functions, and can be minimized in strongly polynomial time. Unfortunately, state-of-the-art algorithms for general submodular minimization are intractable for larger problems. In this paper, we introduce a novel subclass of submodular minimization problems that we call decomposable. Decomposable submodular functions are those that can be represented as sums of concave functions applied to modular functions. We develop an algorithm, SLG, that can efficiently minimize decomposable submodular functions with tens of thousands of variables. Our algorithm exploits recent results in smoothed convex minimization. We apply SLG to synthetic benchmarks and a joint classification-and-segmentation task, and show that it outperforms the state-of-the-art general purpose submodular minimization algorithms by several orders of magnitude.

Motivation & Objective

• To address the computational intractability of general submodular function minimization for large-scale problems in machine learning.
• To identify and exploit a novel subclass of submodular functions—decomposable submodular functions—that admit efficient minimization.
• To develop an algorithm, SLG, that leverages smoothed convex optimization techniques to minimize these functions efficiently.
• To demonstrate the practical scalability and performance of SLG on real-world machine learning tasks, including image segmentation with high-order potentials.

Proposed method

• The paper introduces decomposable submodular functions as sums of concave functions applied to modular functions.
• It formulates the problem using the Lovász extension, which provides a convex continuous relaxation of the discrete submodular function.
• SLG applies smoothed convex minimization techniques to the Lovász extension, enabling efficient optimization via gradient-based methods.
• The algorithm computes subgradients of the Lovász extension using a permutation-based formula involving set differences and function values.
• It uses a smoothing technique that avoids explicit derivative computation by integrating over a Gaussian kernel, enabling efficient subgradient evaluation.
• The method is implemented in MATLAB/Mex and applied to both synthetic and real-world image segmentation tasks with high-order potentials.

Experimental results

Research questions

• RQ1 Can a novel subclass of submodular functions be identified that allows for efficient minimization beyond pairwise potentials?
• RQ2 Can smoothed convex optimization techniques be effectively applied to the Lovász extension to solve large-scale submodular minimization problems?
• RQ3 How does the performance of SLG compare to state-of-the-art general-purpose submodular minimization algorithms on large-scale problems?
• RQ4 Can SLG scale to problems with tens of thousands of variables while maintaining exact or near-exact solutions?

Key findings

• SLG achieved a runtime of 71.4 seconds on a problem with 10,000 variables and 90 concave potentials, while the MinNorm algorithm took 6,900 seconds, representing a 95% reduction in time.
• On a larger image with 40,000 variables, SLG completed in approximately 1,600 seconds, demonstrating scalability beyond the reach of combinatorial algorithms.
• SLG outperformed the MinNorm algorithm by a factor of up to six on synthetic benchmarks, despite using a faster machine and a simpler implementation.
• The algorithm successfully improved segmentation accuracy on boundary regions by incorporating higher-order, concave potentials derived from image regions.
• SLG maintained exact optimality guarantees while achieving practical runtime performance on problems where general-purpose submodular minimization is infeasible.
• The results show that SLG is practical for large-scale machine learning tasks such as joint classification-and-segmentation, where traditional methods fail due to computational complexity.