[Paper Review] Bayesian Optimization With Censored Response Data

Matricon, Théo, Holger H. Hoos+3 more|arXiv (Cornell University)|Oct 7, 2013

Machine Learning and Algorithms20 references22 citations

TL;DR

This paper proposes a Bayesian optimization framework that handles right-censored response data—where only lower bounds on function values are observed—by integrating a novel EM-type algorithm into random forests for imputing censored values. The method enables adaptive censoring in algorithm configuration, significantly improving SMAC’s performance by reducing costly evaluations, achieving up to 126× speedup in median test runtime on real-world solver configurations.

ABSTRACT

Bayesian optimization (BO) aims to minimize a given blackbox function using a model that is updated whenever new evidence about the function becomes available. Here, we address the problem of BO under partially right-censored response data, where in some evaluations we only obtain a lower bound on the function value. The ability to handle such response data allows us to adaptively censor costly function evaluations in minimization problems where the cost of a function evaluation corresponds to the function value. One important application giving rise to such censored data is the runtime-minimizing variant of the algorithm configuration problem: finding settings of a given parametric algorithm that minimize the runtime required for solving problem instances from a given distribution. We demonstrate that terminating slow algorithm runs prematurely and handling the resulting right-censored observations can substantially improve the state of the art in model-based algorithm configuration.

Motivation & Objective

• To address cost-varying blackbox optimization where function evaluation cost equals the function value, enabling resource-aware minimization.
• To handle right-censored data in Bayesian optimization, where only lower bounds on function values are observed due to early termination.
• To improve model-based algorithm configuration (AC) by integrating censored observations from prematurely terminated runs.
• To develop a practical, scalable method for handling high-dimensional, discrete inputs typical in algorithm configuration.
• To demonstrate that adaptive censoring leads to substantial performance gains in real-world configuration scenarios.

Proposed method

• Proposes a cost-monotonic optimization framework where cost c(θ) = f(θ), enabling early termination to save resources.
• Introduces an EM-type algorithm that iteratively imputes censored values by sampling from truncated Gaussian distributions.
• Uses stratified sampling to generate multiple imputations per censored data point, preserving uncertainty in model predictions.
• Adapts random forests to handle censored data by fitting trees on uncensored data and iteratively retraining with imputed censored values.
• Employs a bootstrap-based resampling strategy to maintain data point frequency across trees, tracking resampling counts for imputation.
• Integrates the censored model into SMAC, a state-of-the-art algorithm configuration tool, with adaptive censoring thresholds set via slack factors.

Experimental results

Research questions

• RQ1 Can Bayesian optimization effectively handle right-censored response data where only lower bounds on function values are observed?
• RQ2 How can censored data be modeled in a way that preserves uncertainty and supports active learning in optimization?
• RQ3 Does adaptive censoring—terminating slow runs early and using lower bounds—improve the efficiency of model-based algorithm configuration?
• RQ4 Can the proposed imputation method outperform standard approaches in high-dimensional, discrete input spaces common in algorithm configuration?
• RQ5 What is the optimal slack factor for censoring thresholds in practice, and how does it affect model performance and runtime efficiency?

Key findings

• The modified SMAC with censoring outperformed the original version in all 7 real-world configuration scenarios, with statistical significance in 4.
• The best-performing configuration with censoring achieved a median test runtime up to 126× lower than the non-censored baseline on the SPEAR-SWV problem.
• Censoring with a slack factor of 1.3 yielded the best overall performance across all scenarios, balancing model fit and data acquisition cost.
• On the SPEAR-IBM benchmark, all configurations including censoring achieved identical median performance (1.36×10⁴ seconds), indicating robustness to high variance.
• The method significantly reduced the number of full evaluations required, with the largest gains observed in high-runtime configurations like CPLEX12-CORLAT and SPEAR-SWV.