[Paper Review] Tunability: Importance of Hyperparameters of Machine Learning Algorithms

Philipp Probst, Bernd Bischl+1 more|arXiv (Cornell University)|Feb 26, 2018

Machine Learning and Data ClassificationComputer Science15 references572 citations

TL;DR

The paper formalizes hyperparameter tuning as a statistical problem, defines defaults and tunability measures, and benchmarks 6 algorithms across 38 OpenML datasets to quantify how much tuning improves performance. It provides data-driven optimal defaults, practical tuning spaces, and insights into which hyperparameters matter most.

ABSTRACT

Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this paper is two-fold. Firstly, we formalize the problem of tuning from a statistical point of view, define data-based defaults and suggest general measures quantifying the tunability of hyperparameters of algorithms. Secondly, we conduct a large-scale benchmarking study based on 38 datasets from the OpenML platform and six common machine learning algorithms. We apply our measures to assess the tunability of their parameters. Our results yield default values for hyperparameters and enable users to decide whether it is worth conducting a possibly time consuming tuning strategy, to focus on the most important hyperparameters and to chose adequate hyperparameter spaces for tuning.

Motivation & Objective

• Formalize the problem of hyperparameter tuning from a statistical perspective and define data-based defaults.
• Introduce measures quantifying the tunability of hyperparameters and hyperparameter combinations.
• Develop procedures to estimate tunability and optimal tuning spaces using surrogate models.
• Apply the framework to a large OpenML benchmark to derive practical defaults and insights across algorithms.

Proposed method

• Define R(theta) as the expected risk given hyperparameter configuration theta.
• Propose optimal defaults theta* by minimizing a summary of R^{(j)}(theta) over m datasets.
• Introduce tunability measures d^(j) and d_i^(j) based on differences between default and optimal risk.
• Extend tunability to hyperparameter pairs and joint gains g_i1,i2, including sequential tuning comparisons.
• Define optimal hyperparameter spaces Theta* using dataset-wise quantiles to capture robust tuning ranges.
• Use surrogate models (random forest, etc.) to estimate R^{(j)}(theta) and black-box optimization to find defaults and tunings.
• Experiment with 38 OpenML100 binary classification datasets using six algorithms (glmnet, rpart, kknn, svm, ranger, xgboost) and cross-validation to assess tunability.

Experimental results

Research questions

• RQ1 How should defaults be defined to work well across diverse datasets?
• RQ2 How tunable are common ML algorithms overall and which hyperparameters are most influential?
• RQ3 What is the gain from tuning individual hyperparameters versus combinations?
• RQ4 What are appropriate hyperparameter tuning spaces that capture where performance improves across datasets?
• RQ5 How can surrogate models help estimate tunability and guide automatic tuning?

Key findings

• Optimal defaults significantly improve performance over software defaults for several algorithms, with varying tunability across methods.
• glmnet and svm show higher tunability than ranger, with ranger exhibiting the smallest tunability in the study.
• Individual hyperparameters can have substantial tunability (e.g., gamma, lambda for svm; eta, booster for xgboost).
• Joint tuning of hyperparameter pairs often yields greater gains than tuning single parameters alone (e.g., minsplit and minbucket in rpart).
• The defined tuning spaces (via 5th and 95th percentile quantiles) encompass optimal defaults for many datasets, while some package defaults lie outside the robust range.
• Surrogate models (random forest among others) provide reliable estimates of R(theta) to enable efficient tuning decisions.