[Paper Review] Hyper-Parameter Optimization: A Review of Algorithms and Applications

Tong Yu, Hong Zhu|arXiv (Cornell University)|Mar 12, 2020

Advanced Multi-Objective Optimization Algorithms123 references450 citations

TL;DR

This paper surveys hyper-parameter optimization (HPO) for neural networks, detailing key hyper-parameters, search algorithms, trial schedulers, toolkits, and challenges in deep learning contexts.

ABSTRACT

Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.

Motivation & Objective

• Identify and categorize the major hyper-parameters affecting neural network performance and training efficiency.
• Review and compare existing HPO algorithms and their applicability to deep learning.
• Evaluate current HPO services and toolkits for their support of searching strategies and integration with frameworks.
• Highlight practical challenges and directions for HPO in deep learning applications.

Proposed method

• Categorize hyper-parameters into training-related and model-design-related to structure a systematic HPO discussion.
• Analyze and compare optimization algorithms and trial schedulers for efficiency and accuracy in deep learning.
• Survey mainstream HPO toolkits and services, evaluating their features, compatibility with deep-learning frameworks, and extensibility.
• Discuss model evaluation strategies under limited computational resources and compare optimization approaches.

Experimental results

Research questions

• RQ1 What are the most influential hyper-parameters in neural network training and design, and how should their search spaces be defined?
• RQ2 What are the leading HPO algorithms and trial schedulers, and how do they compare in terms of efficiency and accuracy?
• RQ3 What do current HPO tools and services offer in terms of support for state-of-the-art searching methods and framework integration?
• RQ4 What challenges arise when applying HPO to deep learning, and what directions show promise for future research?

Key findings

• Hyper-parameters are systematically categorized into structure-related and training-related, aiding targeted HPO.
• A detailed analysis compares HPO algorithms by accuracy, efficiency, and applicability, clarifying limitations in various scenarios.
• Toolkits and services are contrasted to reveal design choices and target users for closed-source versus open-source options.
• The paper discusses practical problems in applying HPO to deep learning and highlights efficient model evaluation under computational constraints.
• The review encompasses optimization algorithms, search strategies, and the role of AutoML in reducing human effort while demanding substantial computational resources.