[论文解读] Generalization in Transfer Learning

Suzan Ece Ada, Emre Uğur+1 more|arXiv (Cornell University)|Sep 3, 2019

Reinforcement Learning in Robotics参考文献 36被引用 5

一句话总结

本文提出正则化技术——严格剪裁、早停法和最大熵对抗性学习，以提升连续控制任务中迁移强化学习的泛化能力。通过将训练迭代次数视为超参数并使用策略快照，该方法在多种形态和环境动力学下均实现了优越的迁移性能，将泛化范围扩展至超越以往工作的程度。

ABSTRACT

Agents trained with deep reinforcement learning algorithms are capable of performing highly complex tasks including locomotion in continuous environments. We investigate transferring the learning acquired in one task to a set of previously unseen tasks. Generalization and overfitting in deep reinforcement learning are not commonly addressed in current transfer learning research. Conducting a comparative analysis without an intermediate regularization step results in underperforming benchmarks and inaccurate algorithm comparisons due to rudimentary assessments. In this study, we propose regularization techniques in deep reinforcement learning for continuous control through the application of sample elimination, early stopping and maximum entropy regularized adversarial learning. First, the importance of the inclusion of training iteration number to the hyperparameters in deep transfer reinforcement learning will be discussed. Because source task performance is not indicative of the generalization capacity of the algorithm, we start by acknowledging the training iteration number as a hyperparameter. In line with this, we introduce an additional step of resorting to earlier snapshots of policy parameters to prevent overfitting to the source task. Then, to generate robust policies, we discard the samples that lead to overfitting via a method we call strict clipping. Furthermore, we increase the generalization capacity in widely used transfer learning benchmarks by using maximum entropy regularization, different critic methods, and curriculum learning in an adversarial setup. Subsequently, we propose maximum entropy adversarial reinforcement learning to increase the domain randomization. Finally, we evaluate the robustness of these methods on simulated robots in target environments where the morphology of the robot, gravity, and tangential friction coefficient of the environment are altered.

研究动机与目标

• 解决深度强化学习迁移学习中的泛化不足与过拟合问题。
• 证明源任务性能不能可靠反映目标任务的泛化能力。
• 通过将训练迭代次数视为超参数，提升迁移学习的鲁棒性。
• 开发正则化技术，以增强策略在形态和动力学发生变化的未见任务中的迁移能力。
• 将迁移学习基准的泛化范围扩展至超越现有方法的程度。

提出的方法

• 提出严格剪裁PPO（SC-PPO），将PPO剪裁参数降低至极小值，以消除引发过拟合的样本。
• 通过选择早期策略快照实现早停法，防止对源任务的过拟合。
• 在对抗性强化学习设置中应用最大熵正则化，以增强探索能力和鲁棒性。
• 结合课程学习与熵奖励，提升在领域随机化环境中的对抗性策略训练效果。
• 将SC-PPO与早停法及熵正则化对抗训练相结合，提升在具有挑战性的目标任务中的泛化能力。
• 维护一个策略缓冲区，用于存储并评估训练过程中多个策略快照，以实现更优的泛化选择。

实验结果

研究问题

• RQ1 源任务性能是否能可靠预测迁移强化学习中的泛化能力？
• RQ2 将训练迭代次数视为超参数是否能提升迁移学习性能？
• RQ3 严格剪裁与早停法在减少对源任务过拟合方面的有效性如何？
• RQ4 熵正则化对抗性训练在形态与动力学多样的目标任务中提升泛化的程度如何？
• RQ5 正则化技术的组合是否能将迁移学习的泛化范围扩展至超越先前基准的程度？

主要发现

• SC-PPO与早停法显著提升了目标任务性能，即使在高维人形机器人环境中也能有效防止对源任务的过拟合。
• 与RARL相比，该方法将Hopper形态任务的泛化范围从[2.5, 4.75]扩展至[1, 6]。
• 在重力变化场景下，该方法使目标任务中的前向运动在[0.5G, 1.75G]范围内实现，优于基线方法。
• 熵正则化对抗性训练显著提升了在高摩擦（3.5倍）和高重力（1.75G）环境中的性能。
• 通过使用策略缓冲区和基于迭代的超参数选择，可在未见目标任务中实现更高的启动性能。
• SC-PPO、早停法与熵正则化的组合显著增加了从源任务中提取的可泛化策略数量。