[论文解读] Fast Policy Learning through Imitation and Reinforcement

Ching-An Cheng, Xinyan Yan+2 more|arXiv (Cornell University)|May 26, 2018

Reinforcement Learning in Robotics参考文献 31被引用 44

一句话总结

Loki 以少量随机步推行模仿学习与策略梯度强化学习交错，提供更快收敛并能超越次优专家。

ABSTRACT

Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.

研究动机与目标

• Motivate combining imitation learning (IL) and reinforcement learning (RL) to overcome the limitations of each when the expert is suboptimal.
• Provide a unified mirror-descent view of RL and IL as first-order oracle variants.
• Introduce loki, a simple randomized imitation-then-RL algorithm with theoretical guarantees.
• Demonstrate loki’s empirical performance across simulated control tasks.

提出的方法

• Formulate RL and IL as mirror-descent updates with different first-order oracles.
• Derive policy gradient and imitation gradient update rules within a common framework.
• Define a two-phase loki algorithm that first performs K steps of imitation-based updates and then switches to reinforcement-based updates.
• Randomize the switching point K to achieve favorable convergence properties.
• Provide theoretical guarantees showing loki can match direct policy gradient from the expert under proper randomness.

实验结果

研究问题

• RQ1 Can a simple randomized IL-then-RL procedure outperform a suboptimal expert and converge faster than pure RL from scratch?
• RQ2 Does a unified mirror-descent perspective explain both RL and IL algorithms as variations on a single approach?
• RQ3 What are the theoretical guarantees and practical conditions under which loki matches or surpasses expert-based policy optimization?
• RQ4 How does randomizing the imitation phase duration affect convergence and final performance?

主要发现

• Loki achieves faster learning than standard policy gradient methods by leveraging an IL phase followed by RL.
• Properly randomizing the imitation-to-RL switching time yields performance comparable to running policy gradients directly from the expert.
• Loki can outperform a suboptimal expert and converge faster than RL from scratch in several simulated environments.
• The paper provides a unified mirror-descent framework showing RL and IL differ only by the first-order oracle used.