[Paper Review] SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization

Marius Lindauer, Katharina Eggensperger+7 more|arXiv (Cornell University)|Sep 20, 2021

Machine Learning and Data Classification38 references83 citations

TL;DR

SMAC3 is a flexible, open-source Bayesian optimization package that supports multiple use-cases (HPO, CASH, MF, AC) with various facades, surrogates, and acquisition strategies, enabling robust hyperparameter optimization across tasks.

ABSTRACT

Algorithm parameters, in particular hyperparameters of machine learning algorithms, can substantially impact their performance. To support users in determining well-performing hyperparameter configurations for their algorithms, datasets and applications at hand, SMAC3 offers a robust and flexible framework for Bayesian Optimization, which can improve performance within a few evaluations. It offers several facades and pre-sets for typical use cases, such as optimizing hyperparameters, solving low dimensional continuous (artificial) global optimization problems and configuring algorithms to perform well across multiple problem instances. The SMAC3 package is available under a permissive BSD-license at https://github.com/automl/SMAC3.

Motivation & Objective

• Motivate the importance of finding well-performing hyperparameter configurations for ML algorithms across datasets and tasks.
• Provide a flexible BO framework that is robust to task differences by offering multiple use-case facades.
• Facilitate easy adoption through Python and CLI interfaces, with parallelization options and modular design.

Proposed method

• Implements several Bayesian optimization approaches and acquisition functions (e.g., GP with EI, EI per second, logEI, LCB, TS, PI).
• Supports multiple facades (SMAC4BB, SMAC4HPO, SMAC4MF, SMAC4AC) to tailor BO to low-dimensional continuous optimization, CASH, multi-fidelity tasks, and algorithm configuration.
• Uses surrogate models like random forests and Gaussian processes, with initial design via Sobol sequences or random search.
• Integrates multi-fidelity optimization by combining Hyperband with BO (BOHB-like approach) and uses RF surrogates across fidelity levels.
• Enables aggressive racing and imputation of censored data for AC, along with a dedicated TAE (Target Algorithm Evaluator) interface and a CLI.
• Supports parallelization via DASK and filesystem-based independent SMAC3 instances.

Experimental results

Research questions

• RQ1 Can SMAC3 robustly optimize hyperparameters across diverse problem settings using its multiple facades?
• RQ2 How do SMAC3's surrogate models and acquisition strategies perform in low-dimensional HPO, CASH, multi-fidelity, and AC tasks compared to baselines?
• RQ3 What is the empirical performance of SMAC3's multi-fidelity and aggressive racing mechanisms across benchmark problems?
• RQ4 Does SMAC3 provide practical usability benefits through its interfaces (Python and CLI) and parallelization options?
• RQ5 How does SMAC3 compare to other BO tools (e.g., Dragonfly, BOHB, Hyperband) on standard HPO benchmarks?

Key findings

• SMAC3 offers robust performance across diverse HPO tasks by combining different BO approaches and intensification strategies.
• In multi-fidelity settings, SMAC3's RF-based surrogate with Hyperband-style budgeting matches Hyperband early on, excels in the middle, and eventually catches up with pure BO using RFs.
• SMAC3 consistently outperforms Dragonfly in the reported experiments and, in the later phases, also outperforms BOHB on surrogate benchmarks.
• Empirical comparisons include Net Letter, Naval Propulsion, Nas1Shot1-2 benchmarks from HPOBench, with runtime simulated via surrogates.
• The paper positions SMAC3 as a component of AutoML tools (auto-sklearn, Auto-PyTorch) and competitive in BBO challenges.