[Paper Review] Bayesian Optimisation vs. Input Uncertainty Reduction

Juan Ungredda, Michael Pearce+1 more|arXiv (Cornell University)|May 31, 2020

Simulation Techniques and Applications8 references4 citations

TL;DR

This paper proposes Bayesian Information Collection and Optimisation (BICO), a novel Bayesian optimisation framework that dynamically balances simulation runs and real-world data collection to reduce input uncertainty and improve solution quality. By using a value-of-information approach, BICO automatically selects between simulation or data collection at each iteration, achieving performance comparable to the optimal fixed allocation of resources without prior knowledge of the best split.

ABSTRACT

Simulators often require calibration inputs estimated from real world data and the quality of the estimate can significantly affect simulation output. Particularly when performing simulation optimisation to find an optimal solution, the uncertainty in the inputs significantly affects the quality of the found solution. One remedy is to search for the solution that has the best performance on average over the uncertain range of inputs yielding an optimal compromise solution. We consider the more general setting where a user may choose between either running simulations or instead collecting real world data. A user may choose an input and a solution and observe the simulation output, or instead query an external data source improving the input estimate enabling the search for a more focused, less compromised solution. We explicitly examine the trade-off between simulation and real data collection in order to find the optimal solution of the simulator with the true inputs. Using a value of information procedure, we propose a novel unified simulation optimisation procedure called Bayesian Information Collection and Optimisation (BICO) that, in each iteration, automatically determines which of the two actions (running simulations or data collection) is more beneficial. Numerical experiments demonstrate that the proposed algorithm is able to automatically determine an appropriate balance between optimisation and data collection.

Motivation & Objective

• To address the challenge of input uncertainty in simulation optimisation, where inaccurate input parameters degrade solution quality.
• To develop a unified framework that intelligently balances simulation experiments and real-world data collection to improve solution robustness.
• To enable automatic, adaptive allocation of limited resources between simulation and data collection, avoiding suboptimal fixed-budget splits.
• To model and reduce uncertainty in simulator inputs using Bayesian inference and Gaussian processes.
• To evaluate the performance of the proposed method against fixed allocation strategies in both synthetic and real-world simulation problems.

Proposed method

• BICO uses a Bayesian framework to model both the simulator response and the uncertainty in input parameters using Gaussian processes.
• It applies a value-of-information criterion to compute the expected improvement in solution quality from either running a simulation or collecting real-world data.
• At each iteration, BICO selects the action—simulation or data collection—that maximises the expected improvement in the optimal solution.
• The method employs a joint surrogate model over the solution space and input parameter space, updating beliefs using sequential Bayesian inference.
• It uses a knowledge gradient-inspired acquisition function that accounts for both the uncertainty in the simulator output and the uncertainty in input parameters.
• The algorithm is implemented with efficient computation of expected improvement under input uncertainty, enabling real-time decision-making.

Experimental results

Research questions

• RQ1 Can a unified Bayesian optimisation framework automatically balance simulation and data collection to improve solution quality under input uncertainty?
• RQ2 How does the performance of BICO compare to fixed-budget allocation strategies that pre-allocate resources to data collection?
• RQ3 What is the optimal trade-off between simulation effort and data collection in terms of solution quality and resource efficiency?
• RQ4 Does BICO adaptively allocate more resources to data collection when input uncertainty is high, and to simulation when the input is well-estimated?
• RQ5 Can BICO achieve near-optimal performance without prior knowledge of the best allocation ratio between data and simulation?

Key findings

• BICO achieves solution performance comparable to the optimal fixed allocation of resources, even though the optimal split is unknown in advance.
• In the one-parameter test case, BICO allocated more effort to data collection than in the two-parameter case, likely due to higher-dimensional input space requiring more model credibility.
• In the newsvendor problem, BICO automatically avoided allocating more than 30% of the budget to data collection, steering clear of the suboptimal range found in fixed-allocation strategies.
• The algorithm demonstrated robust performance across both synthetic and real-world simulation problems, maintaining tight confidence intervals on the optimisation cost.
• BICO's adaptive allocation closely matched the performance of the best pre-specified data collection budget, showing strong practical utility.
• The method effectively reduces input uncertainty while maintaining high-quality simulation optimisation, outperforming naive or fixed allocation approaches.