[Paper Review] Deep Determinantal Point Processes

Mike Gartrell, Elvis Dohmatob+1 more|arXiv (Cornell University)|Nov 17, 2018

3D Shape Modeling and Analysis42 references4 citations

TL;DR

This paper proposes the deep determinantal point process (deep DPP), a neural network-based extension of low-rank DPPs that uses a deep feed-forward network to learn nonlinear item embeddings, enabling improved modeling of complex item interactions and integration of metadata. The method significantly boosts predictive performance on real-world datasets, especially in high-complexity and sparse settings, while retaining efficient DPP inference algorithms.

ABSTRACT

Determinantal point processes (DPPs) have attracted significant attention as an elegant model that is able to capture the balance between quality and diversity within sets. DPPs are parameterized by a positive semi-definite kernel matrix. While DPPs have substantial expressive power, they are fundamentally limited by the parameterization of the kernel matrix and their inability to capture nonlinear interactions between items within sets. We present the deep DPP model as way to address these limitations, by using a deep feed-forward neural network to learn the kernel matrix. In addition to allowing us to capture nonlinear item interactions, the deep DPP also allows easy incorporation of item metadata into DPP learning. Since the learning target is the DPP kernel matrix, the deep DPP allows us to use existing DPP algorithms for efficient learning, sampling, and prediction. Through an evaluation on several real-world datasets, we show experimentally that the deep DPP can provide a considerable improvement in the predictive performance of DPPs, while also outperforming strong baseline models in many cases.

Motivation & Objective

• To overcome the limited expressive power of standard DPPs, which are constrained by linear interactions and fixed-rank kernel matrices.
• To enable modeling of nonlinear relationships between items in a set, which standard DPPs cannot capture due to the multilinear nature of the determinant.
• To allow seamless integration of item-level metadata (e.g., names, descriptions) into the DPP learning process without manual feature engineering.
• To improve predictive performance in recommendation and subset selection tasks, particularly in complex or sparse data regimes.
• To maintain compatibility with existing efficient DPP algorithms for learning, sampling, and prediction by using the learned embedding matrix as the kernel.

Proposed method

• The deep DPP uses a deep feed-forward neural network to parameterize the low-rank DPP kernel matrix, where the final hidden layer outputs item embeddings.
• The kernel matrix is constructed as $\mathbf{L} = \mathbf{Z} \mathbf{Z}^T$, where $\mathbf{Z}$ is the matrix of learned item embeddings from the deep network.
• Nonlinear activation functions in the hidden layers allow the model to capture complex, nonlinear interactions between items in a subset.
• Item metadata is embedded into the input layer of the deep network, enabling end-to-end joint learning of embeddings and metadata representations.
• The model is trained end-to-end via maximum likelihood estimation on observed subsets, leveraging standard DPP learning algorithms.
• Inference, sampling, and prediction remain efficient due to the low-rank structure and existing DPP algorithms, even with deep nonlinearities.

Experimental results

Research questions

• RQ1 Can a deep neural network architecture improve the expressive power of DPPs beyond linear interactions between items in a subset?
• RQ2 How does incorporating item metadata affect the predictive performance of DPPs in low-data or cold-start scenarios?
• RQ3 Does the deep DPP outperform standard low-rank DPPs and strong baselines in real-world subset selection tasks?
• RQ4 To what extent does the deep DPP improve discriminative ability in high-complexity datasets with intricate item interactions?
• RQ5 How do deeper network architectures affect the structure and interpretability of learned item embeddings in the DPP framework?

Key findings

• The deep DPP significantly improves predictive performance over standard low-rank DPPs, especially on complex datasets like Instacart and Belgian, with larger MPR gains observed in higher-complexity settings.
• The deep DPP outperforms the pre-trained DPP (using embeddings from another model) in most cases, demonstrating the value of end-to-end learning of the kernel matrix.
• Incorporating item metadata leads to substantial AUC improvements—up to approximately 0.2—for large baskets in the Instacart-10k dataset, indicating enhanced performance in sparse regions.
• t-SNE visualizations show that deeper networks produce more structured and well-separated item embeddings, suggesting effective learning of nonlinear interactions.
• The deep DPP achieves better discrimination between real and synthetic subsets, as evidenced by higher AUC scores, particularly on complex datasets.
• For datasets with subsets no larger than size $K$, increasing the rank beyond $K$ provides no predictive benefit, confirming the theoretical upper bound on model capacity.