[Paper Review] Emulation of physical processes with Emukit

Andrei Paleyes, Mark Pullin+4 more|arXiv (Cornell University)|Oct 25, 2021

Advanced Multi-Objective Optimization AlgorithmsComputer Science20 references56 citations

TL;DR

Emukit is a highly adaptable Python toolkit that unifies emulation, uncertainty quantification, and decision making, allowing user-provided models and backends for Bayesian optimization, experimental design, and more.

ABSTRACT

Decision making in uncertain scenarios is an ubiquitous challenge in real world systems. Tools to deal with this challenge include simulations to gather information and statistical emulation to quantify uncertainty. The machine learning community has developed a number of methods to facilitate decision making, but so far they are scattered in multiple different toolkits, and generally rely on a fixed backend. In this paper, we present Emukit, a highly adaptable Python toolkit for enriching decision making under uncertainty. Emukit allows users to: (i) use state of the art methods including Bayesian optimization, multi-fidelity emulation, experimental design, Bayesian quadrature and sensitivity analysis; (ii) easily prototype new decision making methods for new problems. Emukit is agnostic to the underlying modeling framework and enables users to use their own custom models. We show how Emukit can be used on three exemplary case studies.

Motivation & Objective

• Motivate the use of statistical emulation to replace or augment slow/deterministic simulators in uncertain decision contexts.
• Provide a flexible, backend-agnostic toolkit to enable rapid prototyping and fair benchmarking of decision-making methods.
• Offer a modular framework for coupling surrogate models with decision processes across various applications (e.g., optimization, experimental design, Bayesian quadrature).
• Demonstrate through case studies how Emukit facilitates uncertainty-aware decisions in real-world physical process modeling.

Proposed method

• Define an abstract decision-making loop that captures the common structure of Bayesian optimization, Bayesian quadrature, experimental design, and sensitivity analysis.
• Implement a modular, backend-agnostic architecture where users can wrap custom models and plug them into Emukit’s decision procedures.
• Provide interfaces to connect surrogate models with decision processes without requiring a fixed backend (e.g., NumPy, GPy, TensorFlow, PyTorch).
• Support iterative outer-loop workflows where next inputs are selected, observations are gathered, and the emulator is updated until convergence or stopping criteria are met.
• Illustrate the approach with practical case studies to show flexibility and ease of use for prototyping new methods and benchmarking.

Experimental results

Research questions

• RQ1 How can a model-agnostic toolkit streamline the integration of emulation and decision making in uncertain settings?
• RQ2 Can Emukit support multiple decision problems (optimization, integration, experimental design) within a single coherent workflow?
• RQ3 Do case studies demonstrate practical efficiency and flexibility when using custom models and backends?
• RQ4 What advantages does a modular, component-based design offer for benchmarking and extending decision-making methods?

Key findings

• Emukit enables efficient, uncertainty-aware decision making by wrapping user-defined models into standard decision loops.
• The toolkit demonstrates successful application across diverse domains, including epidemic modeling, quantum-resonator design, and musical synthesizer reverse engineering.
• Emukit’s backend-agnostic design facilitates fair benchmarking and rapid prototyping by allowing interchangeable components (models, samplers, optimizers).
• Case studies show that Emukit can reduce computational costs and enable flexible experimentation compared to fixed-backend workflows.