[Paper Review] Solving Black-Box Optimization Challenge via Learning Search Space Partition for Local Bayesian Optimization

Mikita Sazanovich, Anastasiya Nikolskaya+2 more|arXiv (Cornell University)|Dec 18, 2020

Machine Learning and Data Classification6 citations

TL;DR

This paper presents SPBOpt, a novel black-box optimization algorithm that enhances local Bayesian optimization through learned search space partitioning. By combining multi-task Bayesian optimization to tune its hyperparameters using both local and remote evaluation data, SPBOpt achieved third place in the NeurIPS 2020 Black-Box Optimization Challenge, outperforming state-of-the-art baselines like TuRBO in the final round.

ABSTRACT

Black-box optimization is one of the vital tasks in machine learning, since it approximates real-world conditions, in that we do not always know all the properties of a given system, up to knowing almost nothing but the results. This paper describes our approach to solving the black-box optimization challenge at NeurIPS 2020 through learning search space partition for local Bayesian optimization. We describe the task of the challenge as well as our algorithm for low budget optimization that we named \ exttt{SPBOpt}. We optimize the hyper-parameters of our algorithm for the competition finals using multi-task Bayesian optimization on results from the first two evaluation settings. Our approach has ranked third in the competition finals.

Motivation & Objective

• To address low-budget black-box optimization in real-world machine learning hyperparameter tuning scenarios.
• To improve sample efficiency and performance of Bayesian optimization in high-dimensional, low-budget settings.
• To develop a method that learns where to search in the hyperparameter space by partitioning it based on prior observations.
• To optimize the SPBOpt algorithm’s own hyperparameters using a meta-optimization approach on both local and remote evaluation data.
• To achieve robust performance across diverse test cases, including synthetic and real-world ML problems, under strict iteration and budget constraints.

Proposed method

• SPBOpt uses an initial sampling phase with Latin hypercube sampling combined with ratio-based sampling to generate diverse starting points.
• It applies local Bayesian optimization with a Gaussian process surrogate model and Expected Improvement acquisition function within a dynamically learned region of interest.
• The algorithm learns a search space partition using a support vector machine (SVM) to identify promising subregions based on historical performance.
• The partitioning model is trained on past evaluation results and updated iteratively to refine the search region.
• Hyperparameters of SPBOpt—including initialization points, kernel type, regularization, and decay—were optimized via multi-task Bayesian optimization across local and remote evaluation settings.
• A final candidate configuration was selected using the Wilcoxon signed-rank test to ensure statistical significance in performance across evaluation runs.

Experimental results

Research questions

• RQ1 Can learned search space partitioning improve the sample efficiency and performance of local Bayesian optimization in low-budget black-box optimization?
• RQ2 How does combining local and remote evaluation data improve the robustness of hyperparameter tuning for an optimization algorithm?
• RQ3 Does a meta-optimization approach for configuring the optimization algorithm itself lead to better generalization on unseen test cases?
• RQ4 How does SPBOpt compare to state-of-the-art baselines like TuRBO and hyperopt in terms of final objective value and stability across different evaluation settings?
• RQ5 To what extent does overfitting to remote evaluation servers affect final competition performance, and how can it be mitigated?

Key findings

• SPBOpt achieved a final score of 92.509 in the NeurIPS 2020 Black-Box Optimization Challenge finals, ranking third overall.
• The algorithm outperformed the best baseline, TuRBO, which scored 88.921 in the finals, demonstrating superior robustness.
• All SPBOpt configurations outperformed standard baselines in local evaluation, with the best configuration (SPBOpt 2) achieving a mean score of 98.898.
• Remote evaluation showed that most configurations scored above 97 points on average, with SPBOpt 5 achieving 97.505 ± 0.117.
• The final selection was based on statistical significance using the Wilcoxon signed-rank test, confirming that SPBOpt 2 was significantly better than alternatives.
• The performance drop from remote to final evaluation suggests that many algorithms overfit to the remote server, while SPBOpt’s meta-tuning approach enhanced generalization.