[Paper Review] Faster Greedy MAP Inference for Determinantal Point Processes

In‐Su Han, Prabhanjan Kambadur+2 more|arXiv (Cornell University)|Mar 9, 2017

Data Management and Algorithms32 references3 citations

TL;DR

This paper proposes faster greedy algorithms for MAP inference in large-scale Determinantal Point Processes (DPPs) by leveraging first-order log-determinant approximations and stochastic trace estimators via LDAS. The method reduces time complexity from O(d⁴) to O(d³) by amortizing linear solves and enabling batch selection, achieving up to 19× speedup over baseline greedy methods while sacrificing only marginal accuracy.

ABSTRACT

Determinantal point processes (DPPs) are popular probabilistic models that arise in many machine learning tasks, where distributions of diverse sets are characterized by matrix determinants. In this paper, we develop fast algorithms to find the most likely configuration (MAP) of large-scale DPPs, which is NP-hard in general. Due to the submodular nature of the MAP objective, greedy algorithms have been used with empirical success. Greedy implementations require computation of log-determinants, matrix inverses or solving linear systems at each iteration. We present faster implementations of the greedy algorithms by utilizing the complementary benefits of two log-determinant approximation schemes: (a) first-order expansions to the matrix log-determinant function and (b) high-order expansions to the scalar log function with stochastic trace estimators. In our experiments, our algorithms are orders of magnitude faster than their competitors, while sacrificing marginal accuracy.

Motivation & Objective

• To address the NP-hard nature of MAP inference in large-scale DPPs, which is computationally expensive due to repeated log-determinant, matrix inverse, or linear system computations.
• To accelerate greedy MAP inference by amortizing costly determinant computations using first-order approximations and efficient linear solvers like conjugate gradient (CG).
• To develop a batch-based greedy algorithm that selects multiple items per iteration, reducing the number of expensive operations while maintaining competitive approximation quality.
• To improve the efficiency of high-order log-determinant approximation via LDAS by sharing random vectors across multiple runs, enhancing speed without significant accuracy loss.
• To empirically validate the proposed algorithms on synthetic and real-world datasets, demonstrating substantial speedups over existing greedy and lazy baselines.

Proposed method

• Utilizes first-order expansions of the matrix log-determinant function to approximate marginal gains in greedy selection, reducing per-iteration cost.
• Employs conjugate gradient (CG) solvers to compute Schur complement updates efficiently, amortizing multiple marginal gain computations across a single linear solve.
• Partitions remaining items into p ≥ 1 clusters and applies first-order approximations within each to improve approximation quality while maintaining computational efficiency.
• Introduces a batch greedy strategy that selects k > 1 elements per iteration, reducing the number of iterations and leveraging LDAS for high-order scalar log approximations.
• Applies LDAS with shared random vectors across multiple runs to boost approximation quality and reduce variance, providing theoretical justification for the improvement.
• Combines CG-based linear solves with LDAS-based stochastic trace estimators to balance speed and accuracy, achieving O(d³) time complexity with p = O(1).

Experimental results

Research questions

• RQ1 Can first-order log-determinant approximations combined with efficient linear solvers reduce the time complexity of greedy DPP MAP inference from O(d⁴) to O(d³)?
• RQ2 How does batch selection with high-order log-determinant approximation via LDAS improve efficiency compared to standard greedy or lazy approaches?
• RQ3 Can sharing random vectors across multiple LDAS runs enhance approximation quality and reduce variance in log-determinant estimation?
• RQ4 To what extent do the proposed algorithms maintain approximation quality while achieving significant speedups on large-scale DPP instances?
• RQ5 How do the proposed algorithms compare in practice to established baselines like Lazy, Softmax, and Double on real-world summarization tasks?

Key findings

• Algorithm 2 achieves up to 19× speedup over the Lazy greedy baseline for d = 40,000, with only a 0.01 approximation ratio loss.
• On real-world matched summarization tasks, Algorithm 2 is 8× faster than Lazy for d = 4,000, while losing only 0.03 in average log-probability ratio.
• For video summarization, Algorithm 1 achieves over 13× speedup compared to Lazy, with F-scores nearly identical and sometimes superior to the baseline.
• The proposed algorithms maintain near-optimal performance in practice, with Softmax being significantly slower than even the Lazy baseline on real-world data.
• The batch strategy using LDAS reduces the number of expensive CG runs by replacing pk CG calls with p LDAS runs, leading to substantial speedups when k > 1.
• Theoretical analysis supports the effectiveness of shared random vectors in LDAS, showing improved approximation quality with reduced variance.