[Paper Review] NUBO: A Transparent Python Package for Bayesian Optimization

Mike Diessner, Kevin J. Wilson+1 more|ArXiv.org|May 11, 2023

Advanced Multi-Objective Optimization Algorithms4 citations

TL;DR

NUBO is a lightweight, transparent Python package for Bayesian optimization that enables efficient, sample-efficient optimization of expensive black-box functions—such as physical experiments and simulators—through Gaussian process surrogate models and acquisition functions. It supports sequential, parallel, and asynchronous optimization with bounded, constrained, or mixed (discrete/continuous) parameter spaces, achieving near-optimal performance with only 826 lines of code, outperforming random and Latin hypercube sampling in a 6D test case.

ABSTRACT

NUBO, short for Newcastle University Bayesian Optimisation, is a Bayesian optimization framework for the optimization of expensive-to-evaluate black-box functions, such as physical experiments and computer simulators. Bayesian optimization is a costefficient optimization strategy that uses surrogate modelling via Gaussian processes to represent an objective function and acquisition functions to guide the selection of candidate points to approximate the global optimum of the objective function. NUBO itself focuses on transparency and user experience to make Bayesian optimization easily accessible to researchers from all disciplines. Clean and understandable code, precise references, and thorough documentation ensure transparency, while user experience is ensured by a modular and flexible design, easy-to-write syntax, and careful selection of Bayesian optimization algorithms. NUBO allows users to tailor Bayesian optimization to their specific problem by writing the optimization loop themselves using the provided building blocks. It supports sequential single-point, parallel multi-point, and asynchronous optimization of bounded, constrained, and/or mixed (discrete and continuous) parameter input spaces. Only algorithms and methods that are extensively tested and validated to perform well are included in NUBO. This ensures that the package remains compact and does not overwhelm the user with an unnecessarily large number of options. The package is written in Python but does not require expert knowledge of Python to optimize your simulators and experiments. NUBO is distributed as open-source software under the BSD 3-Clause license.

Motivation & Objective

• To make Bayesian optimization accessible to researchers across disciplines by emphasizing transparency and user experience.
• To provide a minimal, well-documented, and modular implementation of Bayesian optimization that avoids bloat from unnecessary algorithms.
• To support advanced optimization strategies—sequential, parallel, and asynchronous—within a single, consistent framework.
• To enable researchers to tailor the optimization loop using modular building blocks without requiring deep expertise in statistics or programming.
• To outperform standard sampling methods like random and Latin hypercube sampling in convergence to the global optimum with limited function evaluations.

Proposed method

• NUBO uses Gaussian processes as surrogate models to represent the unknown black-box objective function.
• It employs acquisition functions to guide the selection of new evaluation points, balancing exploration and exploitation.
• The package supports sequential single-point, parallel multi-point, and asynchronous optimization via Monte Carlo approximations of acquisition functions.
• It provides modular sub-packages for Gaussian processes, acquisition functions, optimizers, test functions, and utilities, enabling flexible customization.
• The implementation is designed for readability and maintainability, with precise references and thorough documentation to ensure transparency.
• NUBO is built on PyTorch and GPyTorch for efficient computation and supports bounded, constrained, and mixed discrete-continuous parameter spaces.

Figure 1: Bayesian optimisation applied to a 1-dimensional function with one local and one global maximum. Upper confidence bound is used as the acquisition function. The input space is bounded by $[0,10]$ .

Experimental results

Research questions

• RQ1 How can Bayesian optimization be made more accessible to non-expert researchers in science and engineering?
• RQ2 Can a minimal, transparent, and well-documented implementation of Bayesian optimization outperform existing larger packages in usability and performance?
• RQ3 To what extent does NUBO’s support for parallel and asynchronous optimization improve sample efficiency compared to sequential methods?
• RQ4 How does NUBO’s performance compare to random and Latin hypercube sampling in finding the global optimum of a high-dimensional black-box function?
• RQ5 Can a lightweight codebase (826 lines) effectively support advanced Bayesian optimization features without sacrificing reliability or functionality?

Key findings

• NUBO achieved an approximate solution of 3.3076 after 70 evaluations, very close to the true maximum of 3.32237, demonstrating high sample efficiency.
• The algorithm found the best output at iteration 51, indicating effective exploration and exploitation of the parameter space.
• NUBO outperformed both random sampling and Latin hypercube sampling in terms of convergence speed and final solution quality.
• The package’s minimal codebase of 826 lines—compared to BoTorch’s 27,100 lines—demonstrates that high functionality can be achieved with simplicity and clarity.
• NUBO successfully supports parallel and asynchronous optimization, enabling faster convergence when multiple evaluations can be performed simultaneously.
• The modular design allows users to write custom optimization loops using well-documented, composable components, enhancing flexibility and reproducibility.

Figure 2: NUBO flowchart. Overview of the recommended algorithms for specific problems. Start in yellow, decisions in blue, and recommended algorithm in green.