[論文レビュー] Hyper-Parameter Optimization: A Review of Algorithms and Applications

Tong Yu, Hong Zhu|arXiv (Cornell University)|Mar 12, 2020

Advanced Multi-Objective Optimization Algorithms参考文献 123被引用数 450

ひとこと要約

この論文はニューラルネットワークのハイパーパラメータ最適化（HPO）を概観し、主要なハイパーパラメータ、探索アルゴリズム、トライアルスケジューラ、ツールキット、深層学習文脈における課題を詳述する。

ABSTRACT

Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.

研究の動機と目的

• ニューラルネットワークの性能とトレーニング効率に影響を与える主要なハイパーパラメータを特定し、分類する。
• 既存のHPOアルゴリズムを検証・比較し、それらの深層学習への適用性を評価する。
• 現在のHPOサービスとツールキットが探索戦略のサポートとフレームワークとの統合にどの程度対応しているかを評価する。
• 深層学習アプリケーションにおけるHPOの実践的課題と今後の方向性を明らかにする。）
• method([

提案手法

• ハイパーパラメータを「トレーニング関連」と「モデル設計関連」に分類して、体系的なHPO議論の枠組みを作る。
• 深層学習における効率と精度のために、最適化アルゴリズムとトライアルスケジューラを分析・比較する。
• 主流のHPOツールキットとサービスを調査し、機能、深層学習フレームワークとの互換性、拡張性を評価する。
• 計算資源が限られた状況下でのモデル評価戦略を検討し、最適化手法を比較する。

実験結果

リサーチクエスチョン

• RQ1 What are the most influential hyper-parameters in neural network training and design, and how should their search spaces be defined?
• RQ2 What are the leading HPO algorithms and trial schedulers, and how do they compare in terms of efficiency and accuracy?
• RQ3 What do current HPO tools and services offer in terms of support for state-of-the-art searching methods and framework integration?
• RQ4 What challenges arise when applying HPO to deep learning, and what directions show promise for future research?

主な発見

• Hyper-parameters are systematically categorized into structure-related and training-related, aiding targeted HPO.
• A detailed analysis compares HPO algorithms by accuracy, efficiency, and applicability, clarifying limitations in various scenarios.
• Toolkits and services are contrasted to reveal design choices and target users for closed-source versus open-source options.
• The paper discusses practical problems in applying HPO to deep learning and highlights efficient model evaluation under computational constraints.
• The review encompasses optimization algorithms, search strategies, and the role of AutoML in reducing human effort while demanding substantial computational resources.