[Paper Review] Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates

Katharina Eggensperger, Marius Lindauer+3 more|arXiv (Cornell University)|Mar 30, 2017

Advanced Multi-Objective Optimization Algorithms3 references4 citations

TL;DR

This paper proposes using model-based surrogate benchmarks to efficiently evaluate algorithm configuration (AC) procedures by replacing expensive real-world performance evaluations with fast, learned regression models trained on historical AC runs. The surrogate benchmarks preserve the original hyperparameter space and performance characteristics while reducing evaluation cost by orders of magnitude, enabling scalable and reproducible AC research without costly target algorithm executions.

ABSTRACT

The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. The resulting algorithm configuration (AC) problem has attracted much attention from the machine learning community. However, the proper evaluation of new AC procedures is hindered by two key hurdles. First, AC benchmarks are hard to set up. Second and even more significantly, they are computationally expensive: a single run of an AC procedure involves many costly runs of the target algorithm whose performance is to be optimized in a given AC benchmark scenario. One common workaround is to optimize cheap-to-evaluate artificial benchmark functions (e.g., Branin) instead of actual algorithms; however, these have different properties than realistic AC problems. Here, we propose an alternative benchmarking approach that is similarly cheap to evaluate but much closer to the original AC problem: replacing expensive benchmarks by surrogate benchmarks constructed from AC benchmarks. These surrogate benchmarks approximate the response surface corresponding to true target algorithm performance using a regression model, and the original and surrogate benchmark share the same (hyper-)parameter space. In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures. We show that our surrogate benchmarks capture overall important characteristics of the AC scenarios, such as high- and low-performing regions, from which they were derived, while being much easier to use and orders of magnitude cheaper to evaluate.

Motivation & Objective

• To address the high computational cost and setup complexity of benchmarking algorithm configuration (AC) procedures.
• To enable reproducible and scalable evaluation of new AC methods without requiring access to target algorithms or their execution environments.
• To develop surrogate benchmarks that accurately reflect the performance landscape of real AC problems while being orders of magnitude cheaper to evaluate.
• To provide a practical, reusable alternative to both real benchmarks and artificial test functions for AC research.

Proposed method

• Construct surrogate benchmarks by training empirical performance models (EPMs) on historical runs of AC procedures applied to real target algorithms.
• Use regression models to approximate the response surface of target algorithm performance across hyperparameter settings and problem instances.
• Ensure surrogate benchmarks are syntactically equivalent to original AC benchmarks by preserving the same hyperparameter space, instance sets, and performance metrics.
• Train EPMs using data collected during the execution of existing AC procedures, making the process reusable and cost-effective after the initial offline phase.
• Replace expensive calls to the target algorithm with fast predictions from the trained EPM during AC procedure evaluation.
• Leverage established AC benchmark libraries (e.g., AClib, HPOlib) as sources of training data for EPMs.

Experimental results

Research questions

• RQ1 Can surrogate benchmarks trained on real AC runs accurately replicate the performance characteristics of original AC problems?
• RQ2 To what extent do surrogate benchmarks preserve key features such as high- and low-performing regions in the hyperparameter space?
• RQ3 How much computational cost can be saved by using surrogate benchmarks instead of real target algorithm evaluations?
• RQ4 Can surrogate benchmarks support reliable and reproducible evaluation of new AC procedures across diverse domains like hyperparameter optimization and combinatorial solvers?

Key findings

• Surrogate benchmarks capture essential performance characteristics—such as high- and low-performing regions—of original AC problems with high fidelity.
• The proposed approach reduces evaluation cost by multiple orders of magnitude compared to running the actual target algorithms.
• Surrogate benchmarks are reusable and can be used repeatedly to evaluate new AC procedures without re-running expensive target algorithm executions.
• The surrogate benchmarks are syntactically equivalent to original benchmarks, ensuring compatibility with existing AC evaluation pipelines.
• The method enables efficient, scalable, and reproducible AC research even without access to the original target algorithms or their execution environments.