[论文解读] Faster Improvement Rate Population Based Training

Valentin Dalibard, Max Jaderberg|arXiv (Cornell University)|Sep 28, 2021

Machine Learning and Data Classification参考文献 21被引用 4

一句话总结

本文提出Faster Improvement Rate Population Based Training（FIRE PBT），一种增强版PBT算法，通过基于训练改进速率的新型适应度度量，避免贪婪的超参数选择。通过将种群划分为针对学习率调度不同阶段优化的子种群，FIRE PBT在ImageNet和强化学习基准测试中实现了优于标准PBT的最终性能，并与人工调优的学习率调度表现相当。

ABSTRACT

The successful training of neural networks typically involves careful and time consuming hyperparameter tuning. Population Based Training (PBT) has recently been proposed to automate this process. PBT trains a population of neural networks concurrently, frequently mutating their hyperparameters throughout their training. However, the decision mechanisms of PBT are greedy and favour short-term improvements which can, in some cases, lead to poor long-term performance. This paper presents Faster Improvement Rate PBT (FIRE PBT) which addresses this problem. Our method is guided by an assumption: given two neural networks with similar performance and training with similar hyperparameters, the network showing the faster rate of improvement will lead to a better final performance. Using this, we derive a novel fitness metric and use it to make some of the population members focus on long-term performance. Our experiments show that FIRE PBT is able to outperform PBT on the ImageNet benchmark and match the performance of networks that were trained with a hand-tuned learning rate schedule. We apply FIRE PBT to reinforcement learning tasks and show that it leads to faster learning and higher final performance than both PBT and random hyperparameter search.

研究动机与目标

• 为解决PBT倾向于优先考虑短期性能提升而非长期泛化能力的局限性。
• 开发一种方法，通过强调训练过程中的改进速率，识别出能带来更优最终性能的超参数调度。
• 实现自动化的超参数调优，使其在图像分类和强化学习任务中的性能达到或超过人工调优的学习率调度。
• 通过构建可扩展、可并行化的优化框架，减少对人工调优的依赖，实现超参数调度的自主学习。

提出的方法

• FIRE PBT将种群划分为互不重叠的子种群，每个子种群使用基于验证性能改进速率的适应度函数进行训练。
• 适应度度量评估模型在单位训练时间内性能提升的速度，倾向于选择初始改进曲线更陡峭的模型。
• 子种群使用不同的超参数调度策略，每个子种群专注于学习曲线的不同阶段（例如，高学习率与衰减后的学习率）。
• 该方法使用截断选择器，允许表现较差的个体从同一子种群中表现最佳的同伴处复制权重和超参数。
• 学习率和信任区域大小等超参数通过预设缩放因子的乘法噪声独立进行变异。
• 评估器监控性能，并在固定时间间隔触发利用与探索步骤，确保持续适应而不中断训练。

实验结果

研究问题

• RQ1 基于改进速率的适应度度量是否能在超参数优化中优于标准PBT基于即时性能的适应度度量？
• RQ2 FIRE PBT是否能在ImageNet上学习出与人工调优学习率调度性能相当或更优的超参数调度？
• RQ3 与PBT和随机搜索相比，FIRE PBT是否能在强化学习中提升样本效率和最终性能？
• RQ4 FIRE PBT中子种群分解策略是否能实现对复杂、多阶段超参数调度的更好探索？

主要发现

• 在ImageNet基准测试中，FIRE PBT实现了76.5%的测试准确率，优于PBT的71.7%，并匹配人工调优学习率调度的76.6%准确率。
• 在Ant-v1和Humanoid-v1强化学习环境中，FIRE PBT在学习速度和最终性能上均优于PBT和随机搜索，如最高episode回报的分布所示。
• 将改进速率作为适应度度量，成功缓解了PBT过早衰减学习率的倾向，避免了次优的最终性能。
• FIRE PBT在不同环境和超参数配置下表现出鲁棒性，无需额外调优即可保持一致的性能提升。