[Paper Review] Importance of Tuning Hyperparameters of Machine Learning Algorithms

Hilde Weerts, Andreas Mueller+1 more|arXiv (Cornell University)|Jul 15, 2020

Machine Learning and Data ClassificationComputer Science13 references113 citations

TL;DR

The paper introduces a non-inferiority framework to assess hyperparameter tuning importance, proposes a simple default parameter method, and shows many defaults perform non-inferior to tuning on 59 OpenML datasets for RF and SVM.

ABSTRACT

The performance of many machine learning algorithms depends on their hyperparameter settings. The goal of this study is to determine whether it is important to tune a hyperparameter or whether it can be safely set to a default value. We present a methodology to determine the importance of tuning a hyperparameter based on a non-inferiority test and tuning risk: the performance loss that is incurred when a hyperparameter is not tuned, but set to a default value. Because our methods require the notion of a default parameter, we present a simple procedure that can be used to determine reasonable default parameters. We apply our methods in a benchmark study using 59 datasets from OpenML. Our results show that leaving particular hyperparameters at their default value is non-inferior to tuning these hyperparameters. In some cases, leaving the hyperparameter at its default value even outperforms tuning it using a search procedure with a limited number of iterations.

Motivation & Objective

• Assess whether specific hyperparameters require tuning or can be set to reasonable defaults.
• Introduce a simple leave-one-dataset-out default parameter determination procedure.
• Quantify tuning risk and test non-inferiority between fixed-default and fully-tuned configurations.
• Benchmark default values against scikit-learn defaults and prior work across multiple datasets.

Proposed method

• Define a formal setting with a default hyperparameter for each dataset.
• Construct a leave-one-dataset-out approach to select default values from top-performing configurations.
• Compute tuning risk as the difference in risk between fixed-default and tuned settings.
• Use relative tuning risk to assess practical impact on performance.
• Apply a non-inferiority test (non-parametric TOST) to compare fixed vs non-fixed conditions across datasets.
• Evaluate via nested cross-validation with random search within a benchmark of 59 OpenML datasets for RF and SVM.

Experimental results

Research questions

• RQ1 Which hyperparameters are important to tune versus safely fixed at a default value?
• RQ2 Do default hyperparameters determined via the proposed method yield non-inferior performance compared to tuning all hyperparameters?
• RQ3 How does tuning risk vary across datasets and hyperparameters under different performance measures?

Key findings

• Leaving certain hyperparameters at calculated default values is non-inferior to tuning them across many datasets.
• In some cases, fixing defaults even outperforms tuning with a limited random search.
• Default values differ from scikit-learn defaults and prior defaults, and meta-feature dependent defaults are informative for some parameters.
• Tuning risk is often near zero or negative for several hyperparameters, suggesting limited benefit from full tuning given 100 random-search iterations.