[Paper Review] Fast Semidifferential-based Submodular Function Optimization

Rishabh Iyer, Stefanie Jegelka+1 more|arXiv (Cornell University)|Aug 5, 2013

Complexity and Algorithms in Graphs46 references55 citations

TL;DR

This paper introduces a unifying majorize-minimize (MM) framework for submodular optimization using discrete semidifferentials—subgradients and supergradients—enabling efficient, practical algorithms for both unconstrained and constrained submodular minimization and maximization. The approach generalizes and unifies existing greedy and local search methods, achieving state-of-the-art empirical performance and up to 200–500× speedup over exact methods while maintaining strong theoretical guarantees.

ABSTRACT

We present a practical and powerful new framework for both unconstrained and constrained submodular function optimization based on discrete semidifferentials (sub- and super-differentials). The resulting algorithms, which repeatedly compute and then efficiently optimize submodular semigradients, offer new and generalize many old methods for submodular optimization. Our approach, moreover, takes steps towards providing a unifying paradigm applicable to both submodular min- imization and maximization, problems that historically have been treated quite distinctly. The practicality of our algorithms is important since interest in submodularity, owing to its natural and wide applicability, has recently been in ascendance within machine learning. We analyze theoretical properties of our algorithms for minimization and maximization, and show that many state-of-the-art maximization algorithms are special cases. Lastly, we complement our theoretical analyses with supporting empirical experiments.

Motivation & Objective

• To address the scalability and practicality gap in submodular optimization for large-scale machine learning problems.
• To unify traditionally separate approaches to submodular minimization and maximization under a single combinatorial framework.
• To develop efficient, practical algorithms that generalize and improve upon existing greedy and local search methods.
• To provide theoretical bounds and empirical validation for the proposed semidifferential-based optimization framework.
• To reduce computational cost in submodular optimization by leveraging submodular and superdifferentials without continuous relaxation or rounding steps.

Proposed method

• The framework uses discrete semidifferentials—subgradients and supergradients—derived from the submodular polyhedron and anti-submodular polyhedron structures.
• For maximization, the method applies a majorize-minimize (MM) algorithm that iteratively optimizes a surrogate function based on a chosen subgradient.
• For minimization, a complementary majorize-minimize framework is proposed, leveraging superdifferentials to bound the solution space and reduce candidate minimizers.
• The approach avoids continuous relaxation and rounding by remaining fully combinatorial, unlike prior methods based on Lovász or multilinear extensions.
• The algorithmic framework includes variants using different subgradient selection strategies (e.g., greedy, random, local search), with theoretical connections to known approximation algorithms.
• The method integrates as a preprocessing step in exact minimization algorithms to reduce search space via lattice bounds on minimizers.

Experimental results

Research questions

• RQ1 Can a single combinatorial framework unify submodular minimization and maximization under a common optimization paradigm?
• RQ2 Can discrete semidifferentials be used to design faster, practical algorithms that generalize existing greedy and local search methods?
• RQ3 What are the theoretical approximation guarantees and convergence properties of semidifferential-based optimization for submodular functions?
• RQ4 How does the performance of semidifferential-based algorithms compare to exact and relaxation-based methods in practice?
• RQ5 Is the NP-hardness of optimal subgradient selection a fundamental bottleneck, and can heuristic choices still yield strong empirical results?

Key findings

• The proposed MMax framework with subgradient selection (e.g., DLS, BG, RG, RLS) achieves empirical approximation factors significantly better than theoretical worst-case bounds, often outperforming state-of-the-art greedy algorithms.
• The MMax variants are 200–500 times faster than the exact branch-and-bound method of [14], making them highly scalable for large-scale problems.
• The framework generalizes and subsumes many known submodular maximization algorithms, including greedy and local search methods, as special cases.
• For unconstrained minimization, the framework provides new nontrivial bounds on the lattice of minimizers, reducing the search space and accelerating exact algorithms.
• The NP-hardness of finding the optimal subgradient is established, but heuristic subgradient choices still yield strong empirical performance.
• Empirical results on synthetic and real-world data (e.g., TIMIT speech corpus) confirm the method’s robustness and efficiency across diverse submodular objectives.