[Paper Review] Myopic Bayesian Design of Experiments via Posterior Sampling and Probabilistic Programming

Kirthevasan Kandasamy, Willie Neiswanger+4 more|arXiv (Cornell University)|May 25, 2018

Advanced Bandit Algorithms Research30 references3 citations

TL;DR

This paper proposes Myopic Posterior Sampling (MPS), a general-purpose Bayesian design of experiments (DOE) framework that uses probabilistic programming and posterior sampling to sequentially select experiments for complex, multi-goal objectives. It achieves sublinear regret against both myopic and globally optimal policies under adaptive submodularity and monotonicity conditions, enabling effective, scalable DOE in real-world scientific and industrial applications with minimal prior assumptions.

ABSTRACT

We design a new myopic strategy for a wide class of sequential design of experiment (DOE) problems, where the goal is to collect data in order to to fulfil a certain problem specific goal. Our approach, Myopic Posterior Sampling (MPS), is inspired by the classical posterior (Thompson) sampling algorithm for multi-armed bandits and leverages the flexibility of probabilistic programming and approximate Bayesian inference to address a broad set of problems. Empirically, this general-purpose strategy is competitive with more specialised methods in a wide array of DOE tasks, and more importantly, enables addressing complex DOE goals where no existing method seems applicable. On the theoretical side, we leverage ideas from adaptive submodularity and reinforcement learning to derive conditions under which MPS achieves sublinear regret against natural benchmark policies.

Motivation & Objective

• To develop a flexible, general-purpose framework for sequential design of experiments (DOE) that accommodates diverse scientific and industrial goals.
• To enable practitioners to incorporate domain-specific knowledge via Bayesian models and specify goals through customizable penalty functions.
• To design a computationally efficient, myopic strategy that remains competitive with optimal policies despite not performing long-term planning.
• To theoretically justify the performance of the proposed method by deriving conditions under which it achieves sublinear regret relative to optimal policies.
• To demonstrate empirical competitiveness with specialized methods and applicability to complex DOE tasks where existing approaches fail.

Proposed method

• The method employs posterior sampling (Thompson sampling) to select experiments myopically, drawing from the posterior distribution over model parameters to guide action selection.
• It leverages probabilistic programming to express complex, domain-specific models and inference procedures, enabling flexible and scalable implementation.
• The algorithm is designed to be compatible with both synchronous and asynchronous parallel execution using multiple workers.
• A key component is the use of a penalty function $\lambda$ that quantifies the goal-specific cost, with the objective of minimizing expected $\lambda$ over time.
• Theoretical analysis relies on adaptive submodularity and monotonicity to bound the approximation and estimation errors of the myopic policy.
• The method is evaluated empirically across synthetic and real-world DOE tasks, including multi-objective optimization and active learning, showing strong performance against specialized baselines.

Experimental results

Research questions

• RQ1 Under what conditions does a myopic posterior sampling strategy achieve sublinear regret relative to the globally optimal policy in sequential DOE?
• RQ2 How can a general-purpose Bayesian DOE framework be designed to handle complex, multi-goal objectives without requiring problem-specific algorithmic redesign?
• RQ3 To what extent can probabilistic programming and approximate Bayesian inference enable scalable and flexible implementation of DOE strategies in real-world applications?
• RQ4 Can a myopic policy based on posterior sampling effectively balance exploration and exploitation in settings with temporal dependencies and non-linear models?
• RQ5 How does the performance of the proposed method compare to specialized algorithms in terms of both empirical effectiveness and theoretical guarantees?

Key findings

• The proposed Myopic Posterior Sampling (MPS) algorithm achieves sublinear regret against the myopic optimal policy $\pi_{\rm M}^{\star}$, with regret decaying as $\sqrt{\Psi_n / n}$ under mild regularity conditions.
• Under conditions of monotonicity and adaptive submodularity of the penalty function $\lambda$, MPS achieves a performance guarantee of $ (1 - \gamma) \cdot \mathbb{E}[\mu(D^\star_{\gamma n})] - B\sqrt{\Psi_n / (2n)} $, where $\mu = 1 - \lambda$, showing competitiveness with the globally optimal policy $\pi_{\rm G}^{\star}$.
• Empirical results show that MPS performs competitively with specialized methods across diverse DOE tasks, including multi-objective optimization and active learning, despite its general-purpose formulation.
• The method enables solving complex DOE problems where no existing method is applicable, particularly in settings with multiple, interdependent objectives and expensive experiments.
• Theoretical analysis confirms that MPS controls both estimation error (learning the myopic policy) and approximation error (approximating the globally optimal policy), providing a unified performance bound.
• The framework is scalable and supports parallel execution with mild dependence on the number of workers, making it suitable for large-scale DOE applications.