[Paper Review] Dynamics-Aware Unsupervised Discovery of Skills

Archit Sharma, Shixiang Gu+3 more|arXiv (Cornell University)|Jul 2, 2019

Reinforcement Learning in Robotics68 references75 citations

TL;DR

DADS discovers a continuous, predictable set of skills in an unsupervised way and uses their learned dynamics for zero-shot model-based planning, outperforming strong baselines.

ABSTRACT

Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

Motivation & Objective

• Motivate learning diverse, predictable skills without extrinsic reward to facilitate planning.
• Develop a skill-conditioned policy and a skill-specific transition model that render planning in a latent space feasible.
• Show that continuous skill spaces enable richer, more controllable behaviors than discrete Skill sets.
• Demonstrate zero-shot task solving by planning in the learned latent space using model-based methods.

Proposed method

• Maximize a mutual-information objective I(s′; z | s) to encourage diverse yet predictable skills.
• Learn a skill-conditioned policy π(a|s, z) and a skill-conditioned transition model qφ(s′|s, z).
• Apply a variational lower bound to optimize the mutual-information objective and tighten it via KL divergence terms.
• Compute a tractable intrinsic reward r_z(s, a, s′) that promotes predictability under qφ and diversity across z.
• Use model-based planning in the latent space Z with MPC to compose learned skills for downstream tasks, without additional training.

Experimental results

Research questions

• RQ1 Can unsupervised learning of skills yield a continuous, scalable latent space that is easy to predict and plan over?
• RQ2 Does planning in the latent space of skills enable zero-shot solution to downstream tasks with high-dimensional dynamics?
• RQ3 Are continuous skills more suitable for hierarchical composition and long-horizon planning than discrete skills?
• RQ4 How does skill predictability affect variance of behaviors and downstream planning performance?
• RQ5 How does DADS compare to standard model-based and goal-conditioned RL baselines on navigation and locomotion tasks?

Key findings

• DADS learns a diverse set of low-variance, predictable skills without rewards in MuJoCo locomotion tasks.
• A continuous latent space of skills yields smoother, more interpolable behaviors than discrete skill sets.
• Planning over learned skill dynamics with MPC enables zero-shot task solving that outperforms state-of-the-art model-based RL baselines.
• Hierarchical control using MPPI with DADS skills improves downstream navigation tasks compared to DIAYN-based hierarchies and goal-conditioned RL.
• The continuous-primitives variant outperforms discrete ones in hierarchical composition and downstream task performance.