[Paper Review] CMA-ES for Hyperparameter Optimization of Deep Neural Networks

Ilya Loshchilov, Frank Hutter|arXiv (Cornell University)|Apr 25, 2016

Machine Learning and Data Classification18 references238 citations

TL;DR

The paper demonstrates using CMA-ES, a derivative-free optimizer, to search continuous hyperparameters for deep neural networks and compares its parallel performance against Bayesian optimization methods on MNIST using 30 GPUs. It achieves very low validation errors and shows CMA-ES’s strength in parallel settings.

ABSTRACT

Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.

Motivation & Objective

• Motivate hyperparameter optimization for DNNs beyond grid/random search and Bayesian methods.
• Evaluate CMA-ES as a parallelizable, derivative-free optimization approach for continuous hyperparameters.
• Compare CMA-ES with state-of-the-art Bayesian optimization methods in both sequential and parallel settings.
• Assess practical performance on MNIST CNNs with different training time budgets and batch strategies.

Proposed method

• Describe CMA-ES as an iterative sampler of lambda candidate solutions from a multivariate normal distribution.
• Set the initial sampling distribution to N(0.5, 0.2^2) since variables are in [0,1].
• Evaluate candidates by training DNNs with AdaDelta or Adam and selecting batch strategies; objective is the smallest validation error across epochs within a time budget.
• Use lambda = 30 to leverage 30 GPUs and run parallel evaluations; compare 5-minute and 30-minute training budgets.
• Do not employ noise reduction or surrogate models in this study.
• Compare CMA-ES against GP-based Bayesian optimization (Spearmint with EI and PES) and tree-based methods (TPE, SMAC) under parallel and sequential settings.

Experimental results

Research questions

• RQ1 Can CMA-ES compete with Bayesian optimization for continuous DNN hyperparameters?
• RQ2 How does CMA-ES perform in parallel hyperparameter evaluations vs sequential optimization?
• RQ3 How does CMA-ES compare to Spearmint (EI/PES), SMAC, and TPE in terms of final validation performance?
• RQ4 What is the impact of training budget and batch selection strategy on CMA-ES performance?

Key findings

• CMA-ES steadily improves the best validation error over time across all eight hyperparameter settings.
• In the best case, CMA-ES achieves validation errors below 0.3% with 30 minutes of training.
• With a 5-minute budget, CMA-ES reaches about 0.42% validation error.
• In parallel evaluation on 30 GPUs, CMA-ES produces most solutions with validation error below 0.4%, while internal cost is nearly zero.
• GP-based Bayesian optimization (Spearmint with EI/PES) is slower due to cubic GP costs, especially as the number of evaluations grows.
• TPE with Gaussian priors performs well and often matches CMA-ES, while SMAC and EI/PES lag in this noisy, high-dimensional setting.