[Paper Review] Bayesian experimental design using regularized determinantal point processes

Michał Dereziński, Feynman Liang+1 more|arXiv (Cornell University)|Jun 10, 2019

Optimal Experimental Design Methods27 references3 citations

TL;DR

This paper introduces a novel Bayesian experimental design framework using regularized determinantal point processes (DPPs) to select diverse, informative subsets of data for linear regression. By linking Bayesian optimality criteria (A, C, D, V) to DPPs, the authors develop efficient algorithms that achieve (1+ε)-approximate optimal designs when the subset size $k$ is $\Omega\left(\frac{d_{\mathbf{A}}}{\epsilon} + \frac{1}{\epsilon^2}\log\frac{1}{\epsilon}\right)$, significantly improving efficiency and approximation quality over prior methods.

ABSTRACT

In experimental design, we are given $n$ vectors in $d$ dimensions, and our goal is to select $k\ll n$ of them to perform expensive measurements, e.g., to obtain labels/responses, for a linear regression task. Many statistical criteria have been proposed for choosing the optimal design, with popular choices including A- and D-optimality. If prior knowledge is given, typically in the form of a $d imes d$ precision matrix $\mathbf A$, then all of the criteria can be extended to incorporate that information via a Bayesian framework. In this paper, we demonstrate a new fundamental connection between Bayesian experimental design and determinantal point processes, the latter being widely used for sampling diverse subsets of data. We use this connection to develop new efficient algorithms for finding $(1+ε)$-approximations of optimal designs under four optimality criteria: A, C, D and V. Our algorithms can achieve this when the desired subset size $k$ is $Ω(\frac{d_{\mathbf A}}ε + \frac{\log 1/ε}{ε^2})$, where $d_{\mathbf A}\leq d$ is the $\mathbf A$-effective dimension, which can often be much smaller than $d$. Our results offer direct improvements over a number of prior works, for both Bayesian and classical experimental design, in terms of algorithm efficiency, approximation quality, and range of applicable criteria.

Motivation & Objective

• To develop efficient algorithms for Bayesian experimental design under multiple optimality criteria (A, C, D, V) using a novel connection to determinantal point processes.
• To address the challenge of selecting $k \ll n$ informative data points in high-dimensional linear regression with prior knowledge encoded via a precision matrix $\mathbf{A}$.
• To improve upon prior methods in terms of algorithmic efficiency, approximation quality, and applicability to diverse experimental design criteria.

Proposed method

• Leverages a fundamental connection between Bayesian experimental design and regularized determinantal point processes (DPPs), where the DPP kernel is constructed from the data matrix $\mathbf{X}$ and prior precision $\mathbf{A}$.
• Uses a regularized DPP $\mathrm{DPP}_{\mathrm{reg}}^{p}(\mathbf{X}, \mathbf{A})$ with a probability vector $p$ derived from solving a semidefinite program (SDP) to approximate optimal subset selection.
• Employs a first-order convex cone solver (SCS) to efficiently solve the SDP, enabling scalable computation of near-optimal sampling probabilities.
• Introduces a two-stage sampling strategy: first sample from a DPP with learned probabilities, then condition on subset size $k$ via rejection or truncation.
• Applies theoretical guarantees from DPP theory and matrix concentration to bound the effective dimension $d_{\mathbf{A}}$, which governs the required sample size $k$ for $(1+\epsilon)$-approximation.
• Demonstrates that even uniform DPP sampling (without SDP) improves over baseline uniform sampling, and that SDP-optimized DPPs match or exceed greedy methods in performance.

Experimental results

Research questions

• RQ1 Can a principled connection between Bayesian experimental design and determinantal point processes be established to enable efficient subset selection?
• RQ2 What is the minimal subset size $k$ required to achieve a $(1+\epsilon)$-approximation for Bayesian experimental design under A, C, D, and V criteria?
• RQ3 How does the $\mathbf{A}$-effective dimension $d_{\mathbf{A}}$ influence the sample complexity and approximation guarantees?
• RQ4 Can regularized DPPs be used to design algorithms that outperform greedy and uniform sampling strategies in practice?
• RQ5 Is the theoretical upper bound on $\mathrm{OPT}_k$ tight in practice, as suggested by the ratio $f_{\mathbf{A}}(\mathbf{X}_S^\top\mathbf{X}_S)/f_{\mathbf{A}}(\frac{k}{n}\mathbf{\Sigma}_\mathbf{X})$?

Key findings

• The proposed method with SDP-optimized DPP sampling achieves $(1+\epsilon)$-approximation for all four optimality criteria (A, C, D, V) when $k = \Omega\left(\frac{d_{\mathbf{A}}}{\epsilon} + \frac{1}{\epsilon^2}\log\frac{1}{\epsilon}\right)$.
• The method with SDP outperforms greedy bottom-up and predictive length sampling, matching or slightly exceeding greedy performance while maintaining comparable runtime.
• Even uniform DPP sampling (without SDP) improves over uniform i.i.d. sampling, demonstrating that diversity alone enhances design quality.
• Empirical results show that $\mathrm{OPT}_k$ is within a small constant factor (often below 0.5 ratio) of the baseline $f_{\mathbf{A}}(\frac{k}{n}\mathbf{\Sigma}_\mathbf{X})$, supporting the theoretical upper bound.
• The ratio $f_{\mathbf{A}}(\mathbf{X}_S^\top\mathbf{X}_S)/f_{\mathbf{A}}(\frac{k}{n}\mathbf{\Sigma}_\mathbf{X})$ remains below 0.5 across all tested datasets, indicating strong practical performance.
• The use of SCS for solving the SDP enables fast, provably convergent optimization, making the method scalable to real-world datasets.