[Paper Review] Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings

John D. Co-Reyes, YuXuan Liu+4 more|arXiv (Cornell University)|Jun 7, 2018

Reinforcement Learning in Robotics28 references67 citations

TL;DR

SeCTAR learns a continuous latent space of trajectories using a trajectory-level VAE with a state decoder and a latent-conditioned policy decoder, enabling model-based planning in latent space for long-horizon, sparse-reward tasks.

ABSTRACT

In this work, we take a representation learning perspective on hierarchical reinforcement learning, where the problem of learning lower layers in a hierarchy is transformed into the problem of learning trajectory-level generative models. We show that we can learn continuous latent representations of trajectories, which are effective in solving temporally extended and multi-stage problems. Our proposed model, SeCTAR, draws inspiration from variational autoencoders, and learns latent representations of trajectories. A key component of this method is to learn both a latent-conditioned policy and a latent-conditioned model which are consistent with each other. Given the same latent, the policy generates a trajectory which should match the trajectory predicted by the model. This model provides a built-in prediction mechanism, by predicting the outcome of closed loop policy behavior. We propose a novel algorithm for performing hierarchical RL with this model, combining model-based planning in the learned latent space with an unsupervised exploration objective. We show that our model is effective at reasoning over long horizons with sparse rewards for several simulated tasks, outperforming standard reinforcement learning methods and prior methods for hierarchical reasoning, model-based planning, and exploration.

Motivation & Objective

• Motivate representation learning for hierarchical RL by modeling trajectories rather than primitive actions.
• Propose a continuous latent space of skills to enable temporally extended, reusable behaviors.
• Develop a two-headed decoder framework (state decoder and policy decoder) to ensure consistency and enable planning.
• Integrate model-based planning in latent space with an unsupervised exploration objective to tackle sparse rewards.

Proposed method

• Extend variational autoencoder framing to trajectories with a trajectory encoder q_phi(z|tau).
• Use a state decoder p_theta_SD(tau|z) to generate trajectories from latent z.
• Introduce a policy decoder p_theta_PD(a|s,z) that executes in the environment to realize the latent trajectory.
• Enforce consistency between decoders by minimizing KL(p_theta_PD(tau|z) || p_theta_SD(tau|z)) while maximizing ELBO.
• Train encoders/decoders with recurrent networks for the state trajectory, and a feedforward network for the policy decoder.
• Plan in latent space using model predictive control with the state decoder as a predictive model of closed-loop behavior.

Experimental results

Research questions

• RQ1 Can a continuous latent space of trajectories be learned without hand-specified subgoals or discrete skills?
• RQ2 Does joint training of a trajectory-level VAE with a latent-conditioned policy enable reliable planning over long horizons?
• RQ3 Can model-based planning in latent space, aided by an entropy-based exploration objective, improve performance on sparse-reward tasks?
• RQ4 Does the state decoder provide meaningful predictions of outcomes for high-level latent actions?
• RQ5 How does SeCTAR compare to existing model-free, model-based, and hierarchical RL methods on long-horizon tasks?

Key findings

• SeCTAR enables planning over extended trajectories and outperforms several baselines on long-horizon, sparse-reward tasks.
• The latent-space MPC planner uses the state decoder as a trajectory predictor to select latent actions that maximize rewards.
• Joint training yields a consistent state decoder and policy decoder, enabling closed-loop planning and better exploration.
• Unsupervised exploration guided by trajectory-marginal entropy improves state-space coverage and exploration quality.
• Interpolations in latent space produce coherent trajectories, indicating meaningful, generalizable latent representations.
• SeCTAR achieves higher performance and sample efficiency than TRPO, A3C, VIME, FeUdal Networks, and option-critic on the tested tasks.