[Paper Review] Input Warping for Bayesian Optimization of Non-stationary Functions

Jasper Snoek, Kevin Swersky+2 more|arXiv (Cornell University)|Feb 5, 2014

Gaussian Processes and Bayesian Inference35 references100 citations

TL;DR

This paper proposes input warping using the Beta CDF to model non-stationary functions in Bayesian optimization, enabling Gaussian processes to adapt to input-dependent length scales. The method automatically learns bijective transformations of the input space, significantly improving optimization performance on non-stationary problems like hyperparameter tuning, outperforming prior state-of-the-art methods in convergence speed and solution quality.

ABSTRACT

Bayesian optimization has proven to be a highly effective methodology for the global optimization of unknown, expensive and multimodal functions. The ability to accurately model distributions over functions is critical to the effectiveness of Bayesian optimization. Although Gaussian processes provide a flexible prior over functions which can be queried efficiently, there are various classes of functions that remain difficult to model. One of the most frequently occurring of these is the class of non-stationary functions. The optimization of the hyperparameters of machine learning algorithms is a problem domain in which parameters are often manually transformed a priori, for example by optimizing in "log-space," to mitigate the effects of spatially-varying length scale. We develop a methodology for automatically learning a wide family of bijective transformations or warpings of the input space using the Beta cumulative distribution function. We further extend the warping framework to multi-task Bayesian optimization so that multiple tasks can be warped into a jointly stationary space. On a set of challenging benchmark optimization tasks, we observe that the inclusion of warping greatly improves on the state-of-the-art, producing better results faster and more reliably.

Motivation & Objective

• Address the limitation of stationary Gaussian processes in modeling non-stationary functions common in real-world Bayesian optimization problems.
• Develop an automatic, computationally efficient method to learn input-space transformations that mitigate non-stationarity effects.
• Extend the warping framework to multi-task Bayesian optimization to enable shared, jointly stationary representations across tasks.
• Improve optimization reliability and speed by capturing spatially varying length scales in objective functions.
• Enable interpretable post-hoc analysis of learned non-stationarities to gain insights into parameter space structure.

Proposed method

• Apply bijective warping to each input dimension using the cumulative distribution function (CDF) of the Beta distribution to model non-linear, input-dependent transformations.
• Marginalize over the shape parameters of the Beta distribution to learn the warping function in a Bayesian manner.
• Integrate the warping into the kernel function of a Gaussian process to allow flexible, non-stationary modeling of the objective function.
• Extend the framework to multi-task Bayesian optimization by learning a shared, warped input space where all tasks are jointly stationary.
• Use Markov Chain Monte Carlo (MCMC) sampling to infer the posterior distribution over warping parameters and hyperparameters.
• Leverage the interpretability of the Beta CDF to analyze learned transformations and gain insights into parameter sensitivity.

Experimental results

Research questions

• RQ1 Can input warping with the Beta CDF effectively model non-stationary functions in Bayesian optimization, especially when length scales vary across input space?
• RQ2 How does input warping improve optimization performance compared to standard stationary Gaussian process models on non-stationary benchmark problems?
• RQ3 Can the warping framework be successfully extended to multi-task Bayesian optimization to improve transfer learning across related tasks?
• RQ4 Do the learned warping functions align with domain-specific intuition about parameter sensitivity and model behavior?
• RQ5 To what extent does modeling non-stationarity reduce the number of function evaluations required to reach optimal solutions?

Key findings

• Input warping significantly outperforms the state-of-the-art Bayesian optimization method of Snoek et al. (2012) on continuous hyperparameter optimization problems, converging faster and to better solutions.
• On the HPOLib benchmark, the proposed method achieves superior performance across all continuous-valued hyperparameter tuning tasks, demonstrating consistent gains.
• In multi-task Bayesian optimization, warped MTBO outperforms non-warped MTBO by enabling better transfer learning, with faster convergence on target tasks like MNIST after pre-training on USPS.
• The learned warping functions align with domain intuition—for example, higher L2 regularization is favored on smaller datasets like USPS, reflecting data efficiency trade-offs.
• Post-hoc analysis of learned warpings revealed counterintuitive insights in deep learning hyperparameter search, challenging conventional design heuristics.
• The method improves both convergence speed and reliability, with warped Bayesian optimization consistently escaping local minima where standard methods fail.