[Paper Review] Optimization of High-dimensional Simulation Models Using Synthetic Data

Thomas Bartz–Beielstein, Eva Bartz+2 more|arXiv (Cornell University)|Sep 6, 2020

Simulation Techniques and Applications7 references4 citations

TL;DR

This paper presents BuBSim, a discrete-event simulation model for hospital resource planning under high uncertainty, using synthetic data and surrogate-model-based optimization (SMBO) to calibrate over 30 parameters. By combining Poisson-distributed infection events with Gamma-distributed durations and optimizing via SPOT, the method reduces prediction error by 25% (RMSE from 9.16 to 6.81), enabling robust best- and worst-case scenario analysis even with limited real-world data during the COVID-19 pandemic.

ABSTRACT

Simulation models are valuable tools for resource usage estimation and capacity planning. In many situations, reliable data is not available. We introduce the BuB simulator, which requires only the specification of plausible intervals for the simulation parameters. By performing a surrogate-model based optimization, improved simulation model parameters can be determined. Furthermore, a detailed statistical analysis can be performed, which allows deep insights into the most important model parameters and their interactions. This information can be used to screen the parameters that should be further investigated. To exemplify our approach, a capacity and resource planning task for a hospital was simulated and optimized. The study explicitly covers difficulties caused by the COVID-19 pandemic. It can be shown, that even if only limited real-world data is available, the BuB simulator can be beneficially used to consider worst- and best-case scenarios. The BuB simulator can be extended in many ways, e.g., by adding further resources (personal protection equipment, staff, pharmaceuticals) or by specifying several cohorts (based on age, health status, etc.). Keywords: Synthetic data, discrete-event simulation, surrogate-model-based optimization, COVID-19, machine learning, artificial intelligence, hospital resource planning, prediction tool, capacity planning.

Motivation & Objective

• To address the challenge of limited real-world data in hospital capacity planning during the COVID-19 pandemic.
• To develop a simulation framework that operates effectively with only plausible parameter intervals, not requiring historical data.
• To optimize over 30 simulation parameters using surrogate-model-based optimization (SMBO) to improve prediction accuracy.
• To enable sensitivity analysis and scenario exploration (best-case/worst-case) for decision support in health departments.
• To create a flexible, extensible simulator that can incorporate additional resources and patient cohorts.

Proposed method

• The BuBSim model uses discrete-event simulation (DES) to model patient flows through hospital states (e.g., infected, hospitalized, ICU, recovered), with state transitions governed by time durations and probabilities.
• Infection arrival times are modeled using a Poisson process with a daily mean of four new infections over a 91-day period.
• Duration of each state is modeled using a Gamma distribution, with shape and scale parameters treated as optimization variables.
• Surrogate-model-based optimization (SMBO) via the SPOT toolbox is used to calibrate over 30 model parameters by minimizing the root mean square error (RMSE) between simulated and target outcomes.
• Sensitivity analysis is performed using linear regression and regression trees to identify key parameters influencing model error.
• Contour plots visualize interactions between critical parameters (e.g., shape parameter of Gamma distribution and time to hospitalization), supporting parameter space exploration.

Experimental results

Research questions

• RQ1 Can a high-dimensional simulation model be effectively optimized when only plausible intervals for parameters are available, and no real-world data exists?
• RQ2 How does surrogate-model-based optimization (SMBO) improve the accuracy of hospital resource predictions in the absence of empirical data?
• RQ3 Which model parameters are most influential in determining prediction error for ICU and hospital bed demand?
• RQ4 To what extent can synthetic data and simulation enable robust best-case and worst-case scenario analysis for pandemic planning?
• RQ5 How can sensitivity analysis and visualization tools support the interpretation of complex simulation models and guide parameter tuning?

Key findings

• The SMBO-optimized BuBSim model reduced the root mean square error (RMSE) of bed demand predictions by 25.6% (from 9.16 to 6.81), significantly improving accuracy.
• The regression analysis identified that reducing the average days until hospitalization (x₁) and increasing the recovery duration (x₂) would decrease model error.
• The shape parameter of the Gamma distribution (x₁₆) was found to be a critical factor, with higher values improving model fit.
• The percentage of patients requiring ICU ventilation upon hospital arrival (x₁₉) was identified as a key parameter, with higher values reducing error.
• Regression tree analysis confirmed the same four parameters (x₁, x₂, x₁₆, x₁₉) as most important, reinforcing the robustness of the sensitivity analysis.
• Contour plots visualized the trade-off between the shape parameter (x₁₆) and time to hospitalization (x₁), showing that optimal performance occurs when x₁₆ is high and x₁ is low.