[Paper Review] Fast Policy Learning through Imitation and Reinforcement

Ching-An Cheng, Xinyan Yan+2 more|arXiv (Cornell University)|May 26, 2018

Reinforcement Learning in RoboticsComputer Science31 references44 citations

TL;DR

Loki interleaves imitation learning for a few random steps with policy-gradient reinforcement learning, providing faster convergence and the ability to surpass suboptimal experts.

ABSTRACT

Imitation learning (IL) consists of a set of tools that leverage expert demonstrations to quickly learn policies. However, if the expert is suboptimal, IL can yield policies with inferior performance compared to reinforcement learning (RL). In this paper, we aim to provide an algorithm that combines the best aspects of RL and IL. We accomplish this by formulating several popular RL and IL algorithms in a common mirror descent framework, showing that these algorithms can be viewed as a variation on a single approach. We then propose LOKI, a strategy for policy learning that first performs a small but random number of IL iterations before switching to a policy gradient RL method. We show that if the switching time is properly randomized, LOKI can learn to outperform a suboptimal expert and converge faster than running policy gradient from scratch. Finally, we evaluate the performance of LOKI experimentally in several simulated environments.

Motivation & Objective

• Motivate combining imitation learning (IL) and reinforcement learning (RL) to overcome the limitations of each when the expert is suboptimal.
• Provide a unified mirror-descent view of RL and IL as first-order oracle variants.
• Introduce loki, a simple randomized imitation-then-RL algorithm with theoretical guarantees.
• Demonstrate loki’s empirical performance across simulated control tasks.

Proposed method

• Formulate RL and IL as mirror-descent updates with different first-order oracles.
• Derive policy gradient and imitation gradient update rules within a common framework.
• Define a two-phase loki algorithm that first performs K steps of imitation-based updates and then switches to reinforcement-based updates.
• Randomize the switching point K to achieve favorable convergence properties.
• Provide theoretical guarantees showing loki can match direct policy gradient from the expert under proper randomness.

Experimental results

Research questions

• RQ1 Can a simple randomized IL-then-RL procedure outperform a suboptimal expert and converge faster than pure RL from scratch?
• RQ2 Does a unified mirror-descent perspective explain both RL and IL algorithms as variations on a single approach?
• RQ3 What are the theoretical guarantees and practical conditions under which loki matches or surpasses expert-based policy optimization?
• RQ4 How does randomizing the imitation phase duration affect convergence and final performance?

Key findings

• Loki achieves faster learning than standard policy gradient methods by leveraging an IL phase followed by RL.
• Properly randomizing the imitation-to-RL switching time yields performance comparable to running policy gradients directly from the expert.
• Loki can outperform a suboptimal expert and converge faster than RL from scratch in several simulated environments.
• The paper provides a unified mirror-descent framework showing RL and IL differ only by the first-order oracle used.