[论文解读] Diversity is All You Need: Learning Skills without a Reward Function

Benjamin Eysenbach, Abhishek Gupta+2 more|arXiv (Cornell University)|Feb 16, 2018

Reinforcement Learning in Robotics被引用 97

一句话总结

本论文提出 DIAYN，一种无监督方法，通过最大化信息互信息目标与最大熵策略来学习多样技能，使预训练、层级与模仿在没有任务奖励的情况下实现下游任务。

ABSTRACT

Intelligent creatures can explore their environments and learn useful skills without supervision. In this paper, we propose DIAYN ('Diversity is All You Need'), a method for learning useful skills without a reward function. Our proposed method learns skills by maximizing an information theoretic objective using a maximum entropy policy. On a variety of simulated robotic tasks, we show that this simple objective results in the unsupervised emergence of diverse skills, such as walking and jumping. In a number of reinforcement learning benchmark environments, our method is able to learn a skill that solves the benchmark task despite never receiving the true task reward. We show how pretrained skills can provide a good parameter initialization for downstream tasks, and can be composed hierarchically to solve complex, sparse reward tasks. Our results suggest that unsupervised discovery of skills can serve as an effective pretraining mechanism for overcoming challenges of exploration and data efficiency in reinforcement learning.

研究动机与目标

• When reward signals are unavailable or sparse, motivate unsupervised learning of useful skills.
• Propose an information-theoretic objective that yields diverse, discriminable skills expressed as latent-conditioned policies.
• Demonstrate that learned skills can solve benchmark tasks without task rewards and can aid downstream tasks through initialization, hierarchy, and imitation.
• Show stability and empirical robustness of DIAYN across environments and discuss practical benefits for exploration and data efficiency.

提出的方法

• Define a latent variable z representing a skill and train a policy pi_theta(a|s,z) conditioned on z.
• Maximize a variational lower bound of the mutual information between states S and skills Z, plus a term encouraging a high entropy over actions given state, while ensuring discriminability via a discriminator q_phi(z|s).
• Replace the true task reward with a pseudo-reward r_z(s,a)=log q_phi(z|s) - log p(z) and optimize with a maximum entropy RL algorithm (SAC).
• Fix the prior p(z) to be uniform to avoid collapse to a few skills and train a state-conditioned discriminator that looks at all states along trajectories.
• Use a cooperative setup rather than adversarial; train a meta-policy and discriminator jointly to encourage diverse, distinguishable skills.
• Extend DIAYN to hierarchical RL by training a meta-controller to select among learned skills for a fixed horizon, enabling complex tasks with sparse rewards.

实验结果

研究问题

• RQ1 Can unsupervised skill discovery yield diverse, useful policies without any reward signal?
• RQ2 How can an information-theoretic objective promote both discriminability and diversity of skills?
• RQ3 Do learned skills transfer to downstream tasks via pretraining, hierarchical composition, or imitation?
• RQ4 How does DIAYN compare to prior unsupervised skill discovery methods in terms of stability and diversity of learned behaviors?
• RQ5 Can DIAYN facilitate exploration and learning in sparse-reward or high-dimensional environments?

主要发现

• DIAYN learns diverse skills such as running, walking, hopping, flipping, and face plants without any task rewards.
• The learned skills can solve benchmark tasks without receiving the true task reward, and some skills solve tasks in distinct ways.
• Skills can be used to bootstrap downstream tasks via policy initialization, hierarchical RL, and imitation learning, improving sample efficiency.
• The DIAYN objective remains robust across seeds and environments, offering a cooperative training dynamic that avoids instability common in adversarial methods.
• Fixing a uniform prior over skills avoids the Matthew effect seen in VIC, enabling sustained exploration of diverse skills.
• Hierarchical DIAYN enables solving challenging sparse-reward tasks and outperforms competitive baselines in those settings.