[Paper Review] Distributed Submodular Maximization

Baharan Mirzasoleiman, Amin Karbasi+2 more|arXiv (Cornell University)|Nov 3, 2014

Privacy-Preserving Technologies in Data47 references3 citations

TL;DR

This paper proposes GreeDi, a distributed algorithm for submodular function maximization that enables scalable, near-optimal solutions in MapReduce environments. By partitioning data and applying a two-stage greedy-like process with provable approximation guarantees, GreeDi achieves 98% performance of centralized greedy methods on large-scale datasets like tens of millions of points in exemplar clustering and sparse Gaussian processes.

ABSTRACT

Many large-scale machine learning problems--clustering, non-parametric learning, kernel machines, etc.--require selecting a small yet representative subset from a large dataset. Such problems can often be reduced to maximizing a submodular set function subject to various constraints. Classical approaches to submodular optimization require centralized access to the full dataset, which is impractical for truly large-scale problems. In this paper, we consider the problem of submodular function maximization in a distributed fashion. We develop a simple, two-stage protocol GreeDi, that is easily implemented using MapReduce style computations. We theoretically analyze our approach, and show that under certain natural conditions, performance close to the centralized approach can be achieved. We begin with monotone submodular maximization subject to a cardinality constraint, and then extend this approach to obtain approximation guarantees for (not necessarily monotone) submodular maximization subject to more general constraints including matroid or knapsack constraints. In our extensive experiments, we demonstrate the effectiveness of our approach on several applications, including sparse Gaussian process inference and exemplar based clustering on tens of millions of examples using Hadoop.

Motivation & Objective

• Address the scalability limitations of centralized submodular optimization for massive datasets in machine learning.
• Develop a distributed algorithm that maintains strong theoretical approximation guarantees under cardinality, matroid, and knapsack constraints.
• Enable practical deployment on Hadoop and MapReduce-style systems for real-world applications like exemplar clustering and active set selection.
• Theoretical analysis shows that GreeDi achieves performance close to centralized greedy algorithms under natural data partitioning and neighborhood conditions.
• Demonstrate empirical effectiveness on datasets with tens of millions of points, achieving near-optimal results with minimal communication.

Proposed method

• Partition the dataset into m disjoint subsets using random assignment to enable parallel processing.
• Apply a black-box submodular optimization algorithm independently on each partition to compute local solutions.
• Use a two-stage protocol: first, compute local solutions on each machine; second, combine and refine results using a global approximation strategy.
• Leverage the concept of local replaceability and neighborhood density to ensure that feasible solutions in partitions are close to the global optimum.
• Prove approximation guarantees using probabilistic bounds on the size of neighborhoods and feasibility of solutions under constraints.
• Extend the framework to handle non-monotone submodular functions and general constraints like matroids and knapsacks via sampling and partitioning techniques.

Experimental results

Research questions

• RQ1 Can a distributed algorithm achieve approximation guarantees competitive with centralized submodular maximization under cardinality constraints?
• RQ2 How can submodular maximization be effectively parallelized in a MapReduce setting without sacrificing theoretical performance guarantees?
• RQ3 What conditions on data partitioning and neighborhood structure ensure that local solutions are close to the global optimum?
• RQ4 Can the approach be extended to non-monotone submodular functions and complex constraints such as matroids and knapsacks?
• RQ5 How well does the algorithm scale in practice on real-world machine learning tasks with massive datasets?

Key findings

• GreeDi achieves 98% of the objective value of the centralized greedy algorithm in exemplar-based clustering on datasets with tens of millions of points.
• In active set selection for sparse Gaussian processes, the method attains 97% of the centralized solution quality.
• For graph cut finding, the algorithm achieves 90% of the optimal value, demonstrating robustness across diverse applications.
• Theoretical analysis proves that under local replaceability and sufficient neighborhood density, the algorithm maintains a constant-factor approximation guarantee.
• The algorithm's performance is robust to random data partitioning, with high probability ensuring that at least one partition contains a solution close to optimal.
• The method scales efficiently with data size and requires minimal communication, making it suitable for large-scale distributed systems like Hadoop.