[Paper Review] Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking

Laurent Meunier, Herilalaina Rakotoarison+6 more|arXiv (Cornell University)|Oct 8, 2020

Metaheuristic Optimization Algorithms ResearchComputer Science59 references40 citations

TL;DR

The paper introduces OptimSuite, a broad black-box optimization benchmark suite, and ABBO, a general-purpose algorithm selection wizard that performs well across diverse tasks without task-specific tuning.

ABSTRACT

Existing studies in black-box optimization for machine learning suffer from low generalizability, caused by a typically selective choice of problem instances used for training and testing different optimization algorithms. Among other issues, this practice promotes overfitting and poor-performing user guidelines. To address this shortcoming, we propose in this work a benchmark suite, OptimSuite, which covers a broad range of black-box optimization problems, ranging from academic benchmarks to real-world applications, from discrete over numerical to mixed-integer problems, from small to very large-scale problems, from noisy over dynamic to static problems, etc. We demonstrate the advantages of such a broad collection by deriving from it Automated Black Box Optimizer (ABBO), a general-purpose algorithm selection wizard. Using three different types of algorithm selection techniques, ABBO achieves competitive performance on all benchmark suites. It significantly outperforms previous state of the art on some of them, including YABBOB and LSGO. ABBO relies on many high-quality base components. Its excellent performance is obtained without any task-specific parametrization. The OptimSuite benchmark collection, the ABBO wizard and its base solvers have all been merged into the open-source Nevergrad platform, where they are available for reproducible research.

Motivation & Objective

• Address the lack of generalizability in black-box optimization benchmarking due to selective problem instances.
• Create a broad, open benchmark suite (OptimSuite) spanning academic, real-world, discrete, continuous, mixed-integer, noisy, and dynamic problems.
• Develop a general algorithm selection tool (ABBO) that leverages high-level problem features to choose and sequence solvers without task-specific tuning.
• Demonstrate ABBO’s competitive performance across OptimSuite and existing benchmarks like YABBOB and LSGO.

Proposed method

• Construct OptimSuite by integrating multiple benchmarks (e.g., YABBOB, LSGO, MuJoCo, Pyomo, MLDA) into a unified Nevergrad-compatible open platform.
• Design ABBO as a portfolio-based wizard using passive, active (bet-and-run), and chaining selection strategies.
• Use a ask-tell-recommend framework to separate decision points from recommendations, especially in noisy settings.
• Provide a detailed, task-agnostic evaluation protocol with broad problem properties (dimension, variable types, noise presence, budget, parallelism).
• Benchmark ABBO against Shiwa and Nevergrad’s base solvers across OptimSuite variants and existing suites.
• Publish results and code on the Nevergrad dashboard for reproducible research.

Experimental results

Research questions

• RQ1 Can OptimSuite provide a truly broad and unbiased benchmark for black-box optimization across diverse problem settings?
• RQ2 Does ABBO achieve competitive or superior performance across a wide range of benchmarks without task-specific tuning?
• RQ3 How well does ABBO generalize to problems not used in its design or tuning?
• RQ4 What is the impact of chaining and advanced mathematical programming components on ABBO’s performance?
• RQ5 How does ABBO compare to specialized solvers (e.g., LA-MCTS, Shiwa) on established benchmarks like YABBOB and LSGO?

Key findings

• ABBO achieves competitive performance across the broad OptimSuite benchmark and often outperforms previous state-of-the-art methods.
• ABBO shows strong performance on YABBOB and LSGO, and improves over Shiwa due to chaining and stronger local convergence strategies.
• OptimSuite provides open-source, reproducible benchmarking with automatic reruns and dashboards in Nevergrad.
• Cases where newer high-dimensional or noisy bandit-based methods outperform ABBO are acknowledged, indicating ongoing development potential.
• For real-world tasks (MuJoCo, Pyomo, PowerSystems, etc.), ABBO remains robust and competitive, illustrating good generalization.
• ABBO can outperform task-specialized methods like LA-MCTS on several benchmarks while maintaining uniform treatment across tasks.