[Paper Review] Lipschitz-constrained Unsupervised Skill Discovery

Seohong Park, Jongwook Choi+3 more|arXiv (Cornell University)|Feb 2, 2022

Reinforcement Learning in Robotics8 citations

TL;DR

Lipschitz-constrained Skill Discovery (LSD) proposes a novel unsupervised skill discovery method that enforces a Lipschitz constraint on the state representation to encourage dynamic, far-reaching, and diverse skills, outperforming MI-based methods in skill diversity, state space coverage, and zero-shot performance on downstream goal-following tasks across MuJoCo locomotion and manipulation environments.

ABSTRACT

We study the problem of unsupervised skill discovery, whose goal is to learn a set of diverse and useful skills with no external reward. There have been a number of skill discovery methods based on maximizing the mutual information (MI) between skills and states. However, we point out that their MI objectives usually prefer static skills to dynamic ones, which may hinder the application for downstream tasks. To address this issue, we propose Lipschitz-constrained Skill Discovery (LSD), which encourages the agent to discover more diverse, dynamic, and far-reaching skills. Another benefit of LSD is that its learned representation function can be utilized for solving goal-following downstream tasks even in a zero-shot manner - i.e., without further training or complex planning. Through experiments on various MuJoCo robotic locomotion and manipulation environments, we demonstrate that LSD outperforms previous approaches in terms of skill diversity, state space coverage, and performance on seven downstream tasks including the challenging task of following multiple goals on Humanoid. Our code and videos are available at https://shpark.me/projects/lsd/.

Motivation & Objective

• Address the limitation of MI-based skill discovery methods that favor static, low-variation skills due to invariance to scaling and invertible transformations.
• Overcome the need for manual feature engineering or domain-specific priors (e.g., x-y coordinates) to encourage dynamic locomotion or manipulation skills.
• Develop a skill discovery objective that inherently promotes large state-space variations and long-haul behaviors without external rewards.
• Enable zero-shot transfer to downstream goal-following tasks using the learned representation, without additional training or planning.
• Provide a simple, hyperparameter-free method that improves skill diversity and coverage while maintaining strong downstream performance.

Proposed method

• Propose a Lipschitz-constrained objective that maximizes the traveled distance in the state space by constraining the representation function to be Lipschitz-continuous with a bounded constant.
• Formulate the skill discovery objective as a regularized maximization of mutual information between skills and states, with an added Lipschitz penalty to favor large state variations.
• Use a contrastive learning framework to train a policy that maps skill latents to actions, ensuring that different latents lead to distinct and far-reaching state trajectories.
• Leverage the learned state representation for zero-shot goal following by directly regressing the skill latent to the goal state using a simple linear layer.
• Integrate the method with standard RL pipelines by freezing the skill policy and training a hierarchical meta-controller on top for downstream tasks.
• Apply the method to both locomotion (Ant, Humanoid) and manipulation (Fetch) environments with no task-specific reward or intrinsic shaping during pre-training.

Experimental results

Research questions

• RQ1 Can a Lipschitz constraint on the state representation function effectively promote dynamic and diverse skills in unsupervised skill discovery?
• RQ2 Does LSD outperform MI-based baselines in terms of state space coverage and skill diversity without requiring manual feature engineering?
• RQ3 Can the learned representation from LSD enable zero-shot goal following across multiple goals without further training or complex planning?
• RQ4 How does LSD perform on challenging downstream tasks such as multi-goal navigation in Humanoid and object manipulation in Fetch environments?
• RQ5 Is LSD robust and generalizable across different types of robotic control tasks, including both locomotion and manipulation?

Key findings

• LSD achieves the highest state space coverage across five MuJoCo environments, significantly outperforming MI-based baselines such as DIAYN and MDP.
• On the AntMultiGoals and HumanoidMultiGoals tasks, LSD achieves the highest final cumulative rewards, surpassing even methods with task-specific intrinsic rewards or prior knowledge.
• LSD's zero-shot skill selection (without retraining) performs best on AntMultiGoals and remains competitive on other tasks, demonstrating strong generalization.
• In Fetch manipulation tasks, LSD learns to move the target object in the most diverse directions across the x-y plane, as visualized in trajectory maps.
• LSD achieves the best performance in state space coverage (measured in 0.1×0.1 bins) across all three Fetch environments, indicating superior exploration and skill diversity.
• The method requires no additional hyperparameters and is easy to implement, while still outperforming complex baselines with feature engineering or intrinsic rewards.