[Paper Review] Robust Imitation of Diverse Behaviors

Ziyu Wang, Josh Merel+4 more|arXiv (Cornell University)|Jul 10, 2017

Reinforcement Learning in RoboticsComputer Science36 references63 citations

TL;DR

The paper combines a variational autoencoder-based policy with a conditioned GAN-style imitation objective to achieve robust, diverse, one-shot imitation across multiple behaviors in high-dimensional robotic systems.

ABSTRACT

Deep generative models have recently shown great promise in imitation learning for motor control. Given enough data, even supervised approaches can do one-shot imitation learning; however, they are vulnerable to cascading failures when the agent trajectory diverges from the demonstrations. Compared to purely supervised methods, Generative Adversarial Imitation Learning (GAIL) can learn more robust controllers from fewer demonstrations, but is inherently mode-seeking and more difficult to train. In this paper, we show how to combine the favourable aspects of these two approaches. The base of our model is a new type of variational autoencoder on demonstration trajectories that learns semantic policy embeddings. We show that these embeddings can be learned on a 9 DoF Jaco robot arm in reaching tasks, and then smoothly interpolated with a resulting smooth interpolation of reaching behavior. Leveraging these policy representations, we develop a new version of GAIL that (1) is much more robust than the purely-supervised controller, especially with few demonstrations, and (2) avoids mode collapse, capturing many diverse behaviors when GAIL on its own does not. We demonstrate our approach on learning diverse gaits from demonstration on a 2D biped and a 62 DoF 3D humanoid in the MuJoCo physics environment.

Motivation & Objective

• Learn a semantic embedding space for demonstration trajectories using a VAE to enable smooth policy interpolation.
• Combine VAE-based embeddings with a conditioned GAN-like imitation objective to address brittleness and mode collapse.
• Demonstrate robust, diverse behavior imitation from few demonstrations on multi-DOF robots in MuJoCo.
• Enable one-shot imitation by mapping novel trajectories into the learned embedding space.
• Showcase scalability to high-dimensional bodies like a 62-DOF humanoid.

Proposed method

• Train a variational autoencoder on demonstration sequences with a bidirectional LSTM encoder and two decoders (action and state dynamics).
• Decode actions from (state, embedding) with an MLP; decode next states autoregressively via a WaveNet-based state model.
• Use a stochastic VAE to obtain a latent z and minimize reconstruction loss plus KL divergence to p(z).
• Extend GAIL by conditioning the discriminator on VAE embeddings z and marginalize over q(z|x).
• Reward r(x,a|z) = -log(1 - Dψ(x,a|z)) and use TRPO to update the policy, with a fixed VAE prior to stabilize learning.
• Initialize policy around the VAE mean but train a conditioned policy with a Gaussian around μθ(x,z) + μα(x,z) for exploration.

Experimental results

Research questions

• RQ1 Can a VAE-based embedding space capture semantically meaningful, interpolable behavior categories from demonstrations?
• RQ2 Does conditioning GAIL on VAE embeddings reduce mode collapse and improve diversity of learned behaviors?
• RQ3 To what extent can the approach learn robust, diverse policies from a modest number of demonstrations across different bodies (arm, walker, humanoid)?
• RQ4 Can the encoder map novel trajectories into the embedding space for effective one-shot imitation?
• RQ5 How well does the method scale to high-dimensional control problems like a 62-DOF humanoid?

Key findings

• The VAE learns a structured embedding space enabling smooth policy interpolation between demonstrated trajectories.
• Interpolation in latent space corresponds to interpolation in task space for the Jaco arm.
• The conditional discriminator with VAE embeddings produces more robust and diverse imitation than pure BC or vanilla GAIL.
• Adversarial training improves speed-matching and stability for a 2D walker with diverse styles and unseen trajectories.
• The approach yields robust imitation for a high-dimensional humanoid and reduces falling rates compared to non-adaptive baselines.
• Empirical results show clustering in embedding space by movement speed and meaningful transitions between behaviors.