[论文解读] Guarded Policy Optimization with Imperfect Online Demonstrations

Zhenghai Xue, Zhenghao Peng+3 more|arXiv (Cornell University)|Mar 3, 2023

Reinforcement Learning in Robotics被引用 4

一句话总结

该论文提出TS2C，一种新颖的离策略强化学习方法，通过基于轨迹的值估计来决定干预时机，使学生智能体能够在教师-学生框架中超越表现不佳的教师。与基于动作的干预不同，TS2C允许学生在预期回报有前景时偏离教师策略，无论教师质量如何，均能实现更优的样本效率和安全保证。

ABSTRACT

The Teacher-Student Framework (TSF) is a reinforcement learning setting where a teacher agent guards the training of a student agent by intervening and providing online demonstrations. Assuming optimal, the teacher policy has the perfect timing and capability to intervene in the learning process of the student agent, providing safety guarantee and exploration guidance. Nevertheless, in many real-world settings it is expensive or even impossible to obtain a well-performing teacher policy. In this work, we relax the assumption of a well-performing teacher and develop a new method that can incorporate arbitrary teacher policies with modest or inferior performance. We instantiate an Off-Policy Reinforcement Learning algorithm, termed Teacher-Student Shared Control (TS2C), which incorporates teacher intervention based on trajectory-based value estimation. Theoretical analysis validates that the proposed TS2C algorithm attains efficient exploration and substantial safety guarantee without being affected by the teacher's own performance. Experiments on various continuous control tasks show that our method can exploit teacher policies at different performance levels while maintaining a low training cost. Moreover, the student policy surpasses the imperfect teacher policy in terms of higher accumulated reward in held-out testing environments. Code is available at https://metadriverse.github.io/TS2C.

研究动机与目标

• 解决现有教师-学生框架（TSF）方法受限于教师策略性能的局限性。
• 开发一种方法，使学生智能体能够在保持安全性和探索效率的前提下超越表现不佳的教师策略。
• 通过用基于轨迹的值估计替代基于动作相似度的干预，降低对高质量教师示范数据的要求。
• 从理论和实证两方面验证学生性能不再受教师能力上限的约束。
• 在多种环境和不同教师策略来源下实现泛化，保持一致的性能提升。

提出的方法

• 引入基于值的干预机制，仅当学生预期回报低于教师估计值时，教师才进行干预。
• 使用基于轨迹的值估计器评估学生动作的长期回报，以支持明智的干预决策。
• 将教师干预数据与学生经验整合到共享回放缓冲区中，实现离策略学习。
• 应用离策略强化学习算法（如SAC）使用学生生成的经验和教师干预的转移样本联合训练学生策略。
• 设计干预函数以最小化不必要的干预，降低训练成本，同时保持安全性。
• 理论分析表明，TS2C的性能和安全保证具有下界，且独立于教师质量。

Figure 1: Overview of the proposed teacher-student shared control method. Both student and teacher policies are in the training loop and the shared control occurs based on the intervention function.

实验结果

研究问题

• RQ1 在教师-学生框架中，学生策略能否在不被教师性能所限制的情况下超越表现不佳的教师？
• RQ2 将基于动作的干预替换为基于值的干预，是否能提升在线强化学习中的样本效率和安全性？
• RQ3 学生策略的性能如何随教师策略质量的变化而变化？
• RQ4 所提方法能否在不同MuJoCo环境中以及多种教师策略来源下实现泛化？
• RQ5 干预机制对复杂环境中训练成本和策略行为有何影响？

主要发现

• TS2C在所有教师性能水平（高、中、低）下均能在200k训练步内收敛，表现出对教师质量的强鲁棒性。
• 在保留的测试环境中，使用TS2C训练的学生策略获得的累积奖励显著高于教师策略和基线方法（如EGPO、Importance Advising）。
• 与EGPO和Importance Advising相比，TS2C降低了训练成本，因为一旦学生策略掌握控制权，干预率迅速下降。
• 在MetaDrive模拟器中，基于值的干预使学生学会超车行为（蓝色轨迹），而基于动作的干预则导致次优跟随行为（绿色轨迹）。
• 在所有三个测试的MuJoCo环境中，TS2C在教师策略仅中等熟练度时，仍优于SAC、PPO和行为克隆方法。
• 消融实验确认，基于值的干预函数对性能至关重要，因为它使学生即使在偏离教师策略时也能探索高回报动作。

Figure 2: In an autonomous driving scenario, the ego vehicle is the blue one on the left, following the gray vehicle on the right. The upper trajectory is proposed by the student to overtake and the lower trajectory is proposed by the teacher to keep following.