[Paper Review] Third-Person Imitation Learning

Bradly C. Stadie, Pieter Abbeel+1 more|arXiv (Cornell University)|Mar 6, 2017

Reinforcement Learning in Robotics51 references131 citations

TL;DR

The paper proposes an unsupervised third-person imitation learning method that uses domain confusion and a GAN-like setup to learn from demonstrations in a different viewpoint, enabling policy learning in a new domain. It demonstrates success on simple MuJoCo tasks (pointmass, reacher, inverted pendulum) without first-person demonstrations.

ABSTRACT

Reinforcement learning (RL) makes it possible to train agents capable of achieving sophisticated goals in complex and uncertain environments. A key difficulty in reinforcement learning is specifying a reward function for the agent to optimize. Traditionally, imitation learning in RL has been used to overcome this problem. Unfortunately, hitherto imitation learning methods tend to require that demonstrations are supplied in the first-person: the agent is provided with a sequence of states and a specification of the actions that it should have taken. While powerful, this kind of imitation learning is limited by the relatively hard problem of collecting first-person demonstrations. Humans address this problem by learning from third-person demonstrations: they observe other humans perform tasks, infer the task, and accomplish the same task themselves. In this paper, we present a method for unsupervised third-person imitation learning. Here third-person refers to training an agent to correctly achieve a simple goal in a simple environment when it is provided a demonstration of a teacher achieving the same goal but from a different viewpoint; and unsupervised refers to the fact that the agent receives only these third-person demonstrations, and is not provided a correspondence between teacher states and student states. Our methods primary insight is that recent advances from domain confusion can be utilized to yield domain agnostic features which are crucial during the training process. To validate our approach, we report successful experiments on learning from third-person demonstrations in a pointmass domain, a reacher domain, and inverted pendulum.

Motivation & Objective

• Address how to learn from third-person demonstrations without a correspondence between teacher and student states.
• Develop a domain-agnostic representation and reward signal to guide imitation from raw observations.
• Enable policy learning in a novice domain using expert demonstrations from a different domain and viewpoint.

Proposed method

• Formulate third-person RL-GAN where a discriminator distinguishes expert vs non-expert trajectories based on domain-agnostic features.
• Split the discriminator into a feature extractor (D_F) and a classifier (D_R); introduce a domain classifier (D_D) to enforce domain invariance via gradient reversal.
• Use a mutual information-based objective to ensure D_F removes domain-specific information while still enabling discrimination.
• Incorporate a gradient flip (G) to backpropagate domain loss with inverted sign, promoting domain-agnostic features.
• Train the imitator policy π_θ with Trust Region Policy Optimization (TRPO) using the discriminator-based reward −log D_R.
• Extend inputs to multi-time-step observations (o_t, o_{t+n}) to improve discrimination signals.

Experimental results

Research questions

• RQ1 Can third-person imitation learning be solved in simple tasks with observations from differing domains and viewpoints?
• RQ2 Does incorporating domain confusion and multi-time-step inputs improve performance in third-person imitation tasks?
• RQ3 How sensitive is the approach to hyperparameters like the domain confusion weight λ and look-ahead frame count?
• RQ4 How does camera angle difference between expert and novice domains affect learning?
• RQ5 How does the proposed method compare to baselines such as true-reward RL and first-person imitation learning?

Key findings

• The method learns reasonable policies for pointmass, reacher, and inverted pendulum from third-person demonstrations.
• Domain confusion is essential for strong performance across all three tasks; multi-time-step input provides additional gains.
• Feature representations become domain-agnostic, indicating successful third-person learning from raw observations.
• The approach is competitive with first-person imitation and, in some cases, approaches performance with true-reward RL; applying a first-person policy directly to the third-person domain can fail.
• Hyperparameter analysis shows λ needs careful balancing, and a look-ahead window around 4 frames works well across tasks.