[Paper Review] Optimal Learning for Sequential Decisions in Laboratory Experimentation

Kristofer G. Reyes, Warren B. Powell|arXiv (Cornell University)|Apr 11, 2020

Advanced Multi-Objective Optimization Algorithms10 references4 citations

TL;DR

This paper introduces the knowledge gradient policy as an optimal learning framework for sequential laboratory experimentation, enabling scientists to maximize information gain from each experiment by balancing exploration and exploitation using Bayesian belief models. It demonstrates that this approach accelerates discovery in data-scarce domains like drug development and materials science by systematically reducing uncertainty while optimizing for final performance under budget and time constraints.

ABSTRACT

The process of discovery in the physical, biological and medical sciences can be painstakingly slow. Most experiments fail, and the time from initiation of research until a new advance reaches commercial production can span 20 years. This tutorial is aimed to provide experimental scientists with a foundation in the science of making decisions. Using numerical examples drawn from the experiences of the authors, the article describes the fundamental elements of any experimental learning problem. It emphasizes the important role of belief models, which include not only the best estimate of relationships provided by prior research, previous experiments and scientific expertise, but also the uncertainty in these relationships. We introduce the concept of a learning policy, and review the major categories of policies. We then introduce a policy, known as the knowledge gradient, that maximizes the value of information from each experiment. We bring out the importance of reducing uncertainty, and illustrate this process for different belief models.

Motivation & Objective

• To address the slow pace of discovery in laboratory sciences, where experiments are expensive and often fail, by introducing a decision science framework for sequential experimentation.
• To model experimental uncertainty not just in outcomes but in the underlying scientific relationships, using belief models grounded in prior knowledge and data.
• To develop and evaluate learning policies—particularly the knowledge gradient—that maximize the value of information from each experiment.
• To enable risk assessment in experimental design through simulation of policy performance under uncertainty, supporting better budget and resource allocation.
• To guide scientists and robotic systems in making optimal, sequential decisions about which experiments to run, including continuous and discrete parameters.

Proposed method

• Formalizes experimental learning as a sequential decision problem with five core components: state, decisions, outcomes, belief updating, and objectives.
• Uses a Bayesian framework to represent belief models, incorporating prior scientific knowledge and uncertainty in relationships (e.g., temperature vs. yield).
• Introduces the knowledge gradient—a policy that selects the next experiment based on the expected increase in the value of information, maximizing long-term performance.
• Applies lookahead approximations and value function approximations to compute optimal decisions when exact solutions are intractable.
• Employs simulation-based risk assessment by running thousands of policy-driven experiment sequences to evaluate performance distributions and failure rates.
• Supports implementation via protocols for human and robotic scientists, enabling integration with lab automation and real-time decision-making.

Experimental results

Research questions

• RQ1 How can experimental scientists systematically reduce uncertainty in sequential laboratory experiments to accelerate discovery?
• RQ2 What decision policy maximizes the value of information from each experiment in a setting with limited data and high costs?
• RQ3 How can belief models incorporating scientific expertise and prior data be used to guide optimal experimental sequencing?
• RQ4 In what ways can risk in experimental programs be quantitatively assessed before execution using simulation and policy evaluation?
• RQ5 How does the knowledge gradient policy compare to other learning policies in reducing uncertainty and improving final performance?

Key findings

• The knowledge gradient policy consistently outperforms greedy and heuristic approaches by explicitly maximizing the expected improvement in the best design, even with limited data.
• Simulations show that using a robust prior—gained through literature review or modeling—significantly improves final performance, though the cost of prior development must be weighed.
• Experimental risk can be quantitatively assessed via simulation: for example, a policy may achieve a target outcome in 85% of 1,000 simulated runs, with a 15% failure rate under current uncertainty.
• The method enables faster convergence to optimal designs by prioritizing experiments that most reduce uncertainty about the best control settings (e.g., temperature, catalyst).
• Robotic scientists can be integrated into the policy framework, allowing automated, information-driven experimentation that reduces human bias and increases throughput.
• The framework reveals that scientists often overlook critical experimental choices (e.g., new catalysts), favoring easier, incremental changes; the policy helps identify high-impact, non-obvious experiments.