[Paper Review] Imitating Latent Policies from Observation

Ashley D. Edwards, Himanshu Sahni+2 more|arXiv (Cornell University)|May 21, 2018

Reinforcement Learning in RoboticsComputer Science30 references43 citations

TL;DR

ILPO learns latent policies from expert state observations and uses a small amount of environment interaction to map latent actions to real actions, enabling imitation without expert actions and outperforming BC from Observation in several domains.

ABSTRACT

In this paper, we describe a novel approach to imitation learning that infers latent policies directly from state observations. We introduce a method that characterizes the causal effects of latent actions on observations while simultaneously predicting their likelihood. We then outline an action alignment procedure that leverages a small amount of environment interactions to determine a mapping between the latent and real-world actions. We show that this corrected labeling can be used for imitating the observed behavior, even though no expert actions are given. We evaluate our approach within classic control environments and a platform game and demonstrate that it performs better than standard approaches. Code for this work is available at https://github.com/ashedwards/ILPO.

Motivation & Objective

• Infer latent policies from state observations without access to expert actions.
• Learn a latent forward dynamics model to predict next states conditioned on latent actions.
• Develop an action remapping mechanism to align latent actions with real actions using minimal environment interactions.
• Demonstrate ILPO's effectiveness on classic control tasks and a visual platform game against baselines like BCO.

Proposed method

• Train a latent policy network that jointly learns a forward dynamics model G and a prior over latent actions z given states.
• Predict state differences Δt = st+1 − st using G(s, z) and minimize the min over z of ||Δt − G(Ep(st), z)||^2 to capture multimodal transitions.
• Learn a latent policy πω(z|st) by matching the expected next state under the latent distribution to the observed next state.
• Combine with an action remapping network πξ(a|z, Ea(st)) trained using limited environment interactions to map latent actions to real actions.
• Use a two-step process: offline latent policy learning from observations, then ground-truth action remapping via few interactions to enable imitation.

Experimental results

Research questions

• RQ1 Can latent actions inferred from state observations capture the effects of underlying actions without access to expert actions?
• RQ2 How many environment interactions are needed to align latent actions with real actions so that imitation can proceed?
• RQ3 Does learning offline latent dynamics plus a minimal action remapping outperform imitation-from-observation baselines that require more environment data?
• RQ4 Is the approach robust across discrete-action MDPs and a visual platform game with high-dimensional observations?],

Key findings

• ILPO achieves expert-level performance in CartPole and Acrobot with fewer than 100 environment steps.
• ILPO outperforms Behavioral Cloning from Observation on classic control tasks.
• In CoinRun, ILPO improves over BCO but may not reach expert performance in all seeds or levels, reflecting higher difficulty in high-dimensional visual tasks.
• Choosing the latent action set size |Z| near the true action count |A| is beneficial, though the method can learn with other sizes as well.
• The action remapping step requires comparatively few interactions to align latent actions with real actions, reducing sample complexity relative to BCO which learns dynamics online.