[论文解读] Hyper-Parameter Optimization: A Review of Algorithms and Applications

Tong Yu, Hong Zhu|arXiv (Cornell University)|Mar 12, 2020

Advanced Multi-Objective Optimization Algorithms参考文献 123被引用 450

一句话总结

本文综述神经网络的超参数优化（HPO），详述关键超参数、搜索算法、试验调度器、工具包，以及深度学习环境中的挑战。

ABSTRACT

Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.

研究动机与目标

• 识别并分类影响神经网络性能和训练效率的主要超参数。
• 评估并比较现有的HPO算法及其在深度学习中的适用性。
• 评估当前的HPO服务和工具包在搜索策略支持及与框架集成方面的表现。
• 突出深度学习应用中HPO的实际挑战与未来方向。

提出的方法

• 将超参数分为训练相关和模型设计相关，以构建系统化的HPO讨论。
• 分析并比较用于深度学习的优化算法和试验调度器的效率与准确性。
• 调研主流HPO工具包和服务，评估它们的特性、与深度学习框架的兼容性以及可扩展性。
• 在有限计算资源下讨论模型评估策略并比较优化方法。

实验结果

研究问题

• RQ1 在神经网络训练与设计中，哪些超参数最具影响力，应如何定义它们的搜索空间？
• RQ2 领先的HPO算法和试验调度器有哪些？它们在效率和精度方面的比较如何？
• RQ3 当前的HPO工具和服务在支持前沿搜索方法与框架集成方面提供了哪些功能？
• RQ4 在将HPO应用于深度学习时会出现哪些挑战，哪些方向对未来研究有潜在帮助？

主要发现

• 超参数被系统性地分为结构相关和训练相关，有助于有针对性的HPO。
• 对HPO算法在准确性、效率和适用性方面的详细分析，阐明了在不同情境中的局限性。
• 对工具包和服务进行对比，揭示封闭源与开源选项的设计选择与目标用户。
• 本文讨论将HPO应用于深度学习的实际问题，并强调在计算资源受限下的高效模型评估。
• 综述涵盖优化算法、搜索策略，以及AutoML在减少人工努力的同时对大量计算资源的需求。