[Paper Review] Goal-Conditioned Imitation Learning

Yiming Ding|arXiv (Cornell University)|Jun 13, 2019

Reinforcement Learning in Robotics38 references32 citations

TL;DR

The paper introduces goalGAIL, a goal-conditioned Generative Adversarial Imitation Learning method with hindsight relabeling to accelerate learning of goals in robotics, including support for state-only and suboptimal demonstrations, and expert relabeling for data augmentation.

ABSTRACT

Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might require many samples to discover how to reach certain areas of the state-space. In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, we show our method can also be used when the available expert trajectories do not contain the actions, which can leverage kinesthetic or third person demonstration. The code is available at https://sites.google.com/view/goalconditioned-il/.

Motivation & Objective

• Motivate the difficulty of reward design in real-world RL and the need for reward-free, goal-reaching policies.
• Develop a goal-conditioned imitation learning framework that can reach any observed goal efficiently.
• Leverage demonstrations to speed up convergence and generalization beyond the demonstrator.
• Introduce goalGAIL, a GAIL-based algorithm compatible with off-policy training and hindsight relabeling.
• Extend functionality to handle state-only, kinesthetic, and suboptimal expert demonstrations.

Proposed method

• Condition the discriminator on (a, s, g) so the agent is rewarded for producing transitions toward the goal.
• Use a GAIL objective to train a policy via an off-policy algorithm (DDPG) with a combined reward: r = indicator(s'==g) and an annealed log Dψ(a,s,g) term.
• Introduce expert relabeling: relabel expert transitions with future goals s(t+k) to increase supervisory signal from few demonstrations.
• Propose goalGAIL, combining GAIL with Hindsight Experience Replay (HER) to enable learning from goal-conditioned tasks.
• Allow state-only demonstrations by replacing the action with the next state in the discriminator, enabling learning from kinesthetic or third-person demonstrations.
• Provide a novel relabeling scheme for expert trajectories to augment data when demonstrations are scarce.

Experimental results

Research questions

• RQ1 Can goalGAIL accelerate goal-conditioned learning and outperform the demonstrator without explicit reward supervision?
• RQ2 Does expert relabeling improve data efficiency and final performance across goal-conditioned tasks?
• RQ3 Is goalGAIL robust to sub-optimal experts and capable of using state-only demonstrations?
• RQ4 How does state-only demonstration performance compare to full state-action demonstrations in guiding goal-conditioned policies?

Key findings

• goalGAIL consistently converges faster than HER and achieves better final performance than naive goal-conditioned GAIL across four continuous MuJoCo tasks.
• Expert relabeling significantly boosts final performance for BC, BC+HER, and goalGAIL across environments.
• GAIL-based methods show robustness to sub-optimal experts, outperforming pure BC in noisy demonstration settings.
• State-only demonstrations can outperform BC+HER and even full-action GAIL in most tasks, demonstrating flexibility in available supervision.
• goalGAIL outperforms baselines by leveraging demonstrations while still benefiting from hindsight relabeling and off-policy learning.