[论文解读] Efficient Benchmarking of Algorithm Configuration Procedures via Model-Based Surrogates

Katharina Eggensperger, Marius Lindauer+3 more|arXiv (Cornell University)|Mar 30, 2017

Advanced Multi-Objective Optimization Algorithms参考文献 3被引用 4

一句话总结

本文提出使用基于模型的代理基准来高效评估算法配置（AC）过程，通过用快速、学习得到的回归模型替代昂贵的真实世界性能评估。代理基准在保留原始超参数空间和性能特征的同时，将评估成本降低多个数量级，从而实现在无需昂贵的目标算法执行的情况下，可扩展且可复现的AC研究。

ABSTRACT

The optimization of algorithm (hyper-)parameters is crucial for achieving peak performance across a wide range of domains, ranging from deep neural networks to solvers for hard combinatorial problems. The resulting algorithm configuration (AC) problem has attracted much attention from the machine learning community. However, the proper evaluation of new AC procedures is hindered by two key hurdles. First, AC benchmarks are hard to set up. Second and even more significantly, they are computationally expensive: a single run of an AC procedure involves many costly runs of the target algorithm whose performance is to be optimized in a given AC benchmark scenario. One common workaround is to optimize cheap-to-evaluate artificial benchmark functions (e.g., Branin) instead of actual algorithms; however, these have different properties than realistic AC problems. Here, we propose an alternative benchmarking approach that is similarly cheap to evaluate but much closer to the original AC problem: replacing expensive benchmarks by surrogate benchmarks constructed from AC benchmarks. These surrogate benchmarks approximate the response surface corresponding to true target algorithm performance using a regression model, and the original and surrogate benchmark share the same (hyper-)parameter space. In our experiments, we construct and evaluate surrogate benchmarks for hyperparameter optimization as well as for AC problems that involve performance optimization of solvers for hard combinatorial problems, drawing training data from the runs of existing AC procedures. We show that our surrogate benchmarks capture overall important characteristics of the AC scenarios, such as high- and low-performing regions, from which they were derived, while being much easier to use and orders of magnitude cheaper to evaluate.

研究动机与目标

• 为解决基准化算法配置（AC）过程所面临的高计算成本和设置复杂性问题。
• 在无需访问目标算法或其执行环境的情况下，实现新AC方法的可复现且可扩展的评估。
• 开发能准确反映真实AC问题性能分布的代理基准，同时将评估成本降低多个数量级。
• 为AC研究提供一种实用且可重用的替代方案，既优于真实基准，也优于人工测试函数。

提出的方法

• 通过在真实目标算法上应用AC过程的历史运行数据上训练经验性能模型（EPMs），构建代理基准。
• 使用回归模型近似目标算法在不同超参数设置和问题实例下的性能响应曲面。
• 通过保留相同的超参数空间、实例集和性能度量，确保代理基准在语法上与原始AC基准保持一致。
• 利用现有AC过程执行期间收集的数据训练EPMs，使该过程在初始离线阶段后可重复使用且成本低廉。
• 在AC过程评估期间，用训练好的EPM的快速预测替代对目标算法的昂贵调用。
• 利用成熟的AC基准库（如AClib、HPOlib）作为EPM训练数据的来源。

实验结果

研究问题

• RQ1 基于真实AC运行训练的代理基准能否准确复现原始AC问题的性能特征？
• RQ2 代理基准在多大程度上保留了超参数空间中高性能和低性能区域等关键特征？
• RQ3 与真实目标算法评估相比，使用代理基准能节省多少计算成本？
• RQ4 代理基准能否在超参数优化和组合求解器等多样化领域中，支持新AC过程的可靠且可复现的评估？

主要发现

• 代理基准以高保真度捕捉了原始AC问题的关键性能特征，如高性能和低性能区域。
• 与实际运行目标算法相比，所提出方法将评估成本降低了多个数量级。
• 代理基准可重复使用，可多次用于评估新AC过程，而无需重新运行昂贵的目标算法执行。
• 代理基准在语法上与原始基准完全一致，确保与现有AC评估流水线的兼容性。
• 该方法即使在无法访问原始目标算法或其执行环境的情况下，也能实现高效、可扩展且可复现的AC研究。