[Paper Review] Bayesian Optimization with Dimension Scheduling: Application to Biological Systems

Doniyor Ulmasov, Caroline Baroukh+3 more|arXiv (Cornell University)|Nov 17, 2015

Advanced Multi-Objective Optimization Algorithms14 references6 citations

TL;DR

This paper proposes Dimension Scheduling Algorithm (DSA), a Bayesian Optimization method that accelerates convergence by optimizing only a subset of dimensions at each iteration, reducing computational cost. DSA achieves faster convergence and lower objective values than traditional BO in high-dimensional biological model parameter estimation, particularly when simulations are fast and numerous.

ABSTRACT

Bayesian Optimization (BO) is a data-efficient method for global black-box optimization of an expensive-to-evaluate fitness function. BO typically assumes that computation cost of BO is cheap, but experiments are time consuming or costly. In practice, this allows us to optimize ten or fewer critical parameters in up to 1,000 experiments. But experiments may be less expensive than BO methods assume: In some simulation models, we may be able to conduct multiple thousands of experiments in a few hours, and the computational burden of BO is no longer negligible compared to experimentation time. To address this challenge we introduce a new Dimension Scheduling Algorithm (DSA), which reduces the computational burden of BO for many experiments. The key idea is that DSA optimizes the fitness function only along a small set of dimensions at each iteration. This DSA strategy (1) reduces the necessary computation time, (2) finds good solutions faster than the traditional BO method, and (3) can be parallelized straightforwardly. We evaluate the DSA in the context of optimizing parameters of dynamic models of microalgae metabolism and show faster convergence than traditional BO.

Motivation & Objective

• Address the inefficiency of traditional Bayesian Optimization (BO) when experiment evaluation is fast but BO computation is costly, especially in high-dimensional parameter spaces.
• Overcome the cubic scaling of Gaussian Process (GP) inference in BO, which limits scalability to ~1,000 experiments and 10 parameters.
• Enable efficient parameter estimation in dynamic biological models—such as microalgae metabolism—where simulations are fast but require thousands of evaluations.
• Develop a scalable, parallelizable BO framework that maintains model accuracy while drastically reducing per-iteration computational load.
• Improve convergence speed and solution quality in parameter estimation tasks where traditional BO becomes computationally prohibitive due to data size and dimensionality.

Proposed method

• Introduce a Dimension Scheduling Algorithm (DSA) that selects a small, dynamic subset of dimensions at each BO iteration based on a probability distribution reflecting parameter importance.
• Decompose the full data set across multiple GPs, each trained on a subset of dimensions and corresponding observations, reducing per-GP data size and computational cost.
• Optimize the acquisition function only over the selected subset of dimensions at each iteration, significantly reducing the time required for GP prediction and optimization.
• Use a manager process to coordinate multiple worker processes, each maintaining a GP and solving the acquisition function for its assigned dimension subset, enabling straightforward parallelization.
• Distribute training data across GPs such that each GP handles a small, frequently updated subset of the full data, minimizing the number of observations per GP and improving scalability.
• Apply marginal likelihood to prune less relevant GPs and reduce model complexity, improving efficiency without sacrificing solution quality.

Experimental results

Research questions

• RQ1 Can Bayesian Optimization be made scalable for high-dimensional, fast-evaluating simulation models where traditional BO becomes computationally prohibitive?
• RQ2 Does optimizing only a subset of dimensions at each iteration improve convergence speed and solution quality in parameter estimation for dynamic biological systems?
• RQ3 How does the computational overhead of DSA compare to traditional BO when applied to high-dimensional biological models with thousands of simulations?
• RQ4 To what extent can DSA be parallelized, and does this parallelization lead to measurable performance gains in optimization time?
• RQ5 What is the trade-off between using only partial data in GPs and the resulting accuracy and convergence of the optimization process?

Key findings

• DSA reduces average computation time per experiment to approximately one-fifth of traditional BO, as shown in Figure 3, due to smaller GP training sets and reduced dimensionality per iteration.
• DSA achieves a lower best objective value than traditional BO in most cases, with faster convergence observed in Figure 2, particularly in early iterations.
• DSA improves convergence frequency by frequently re-evaluating different dimension subsets, enabling more dynamic exploration of the search space.
• The method enables efficient parameter estimation in high-dimensional microalgae metabolism models where traditional BO fails due to computational scaling issues.
• DSA supports straightforward parallelization by distributing GP and acquisition function optimization across multiple processes, with minimal architectural changes.
• Despite its advantages, DSA occasionally underperforms traditional BO due to data sparsity across multiple GPs, especially when the number of dimensions increases.