[Paper Review] Discovering Diverse Solutions in Deep Reinforcement Learning.

Takayuki Osa, Voot Tangkaratt+1 more|arXiv (Cornell University)|Mar 12, 2021

Reinforcement Learning in Robotics32 references7 citations

TL;DR

This paper proposes a deep reinforcement learning method that trains a policy conditioned on continuous or discrete low-dimensional latent variables to discover an infinite set of diverse solutions for a given task. By enabling data-efficient learning of diverse behaviors, the approach supports robust few-shot adaptation to unseen tasks, demonstrating superior generalization and solution variety in continuous control environments.

ABSTRACT

Reinforcement learning (RL) algorithms are typically limited to learning a single solution of a specified task, even though there often exists diverse solutions to a given task. Compared with learning a single solution, learning a set of diverse solutions is beneficial because diverse solutions enable robust few-shot adaptation and allow the user to select a preferred solution. Although previous studies have showed that diverse behaviors can be modeled with a policy conditioned on latent variables, an approach for modeling an infinite set of diverse solutions with continuous latent variables has not been investigated. In this study, we propose an RL method that can learn infinitely many solutions by training a policy conditioned on a continuous or discrete low-dimensional latent variable. Through continuous control tasks, we demonstrate that our method can learn diverse solutions in a data-efficient manner and that the solutions can be used for few-shot adaptation to solve unseen tasks.

Motivation & Objective

• To address the limitation of standard RL algorithms that learn only a single solution per task, despite the existence of multiple viable behaviors.
• To explore whether continuous latent variables can model an infinite set of diverse solutions in deep RL.
• To enable data-efficient learning of diverse behaviors that support rapid adaptation to new, unseen tasks.
• To develop a scalable and generalizable method for discovering diverse policies without relying on predefined behavior clusters.

Proposed method

• A policy network is conditioned on a low-dimensional continuous or discrete latent variable to generate diverse behaviors.
• The method uses a variational inference framework to train the policy and latent variable distribution jointly, encouraging diversity in the learned behaviors.
• A reward shaping mechanism ensures that diverse behaviors are not only distinct but also effective in solving the task.
• The algorithm optimizes a regularized objective that balances task performance and behavioral diversity through the latent space.
• The latent variables are sampled during inference to generate new, diverse solutions on demand.
• The approach is evaluated on continuous control benchmarks such as MuJoCo environments, demonstrating scalability and efficiency.

Experimental results

Research questions

• RQ1 Can a deep RL algorithm learn an infinite set of diverse solutions using continuous latent variables?
• RQ2 How does the proposed method compare to existing approaches in terms of data efficiency and solution diversity?
• RQ3 To what extent can the learned diverse policies enable few-shot adaptation to unseen tasks?
• RQ4 Does the use of continuous latent variables lead to better generalization and behavioral variety than discrete or fixed behavior priors?

Key findings

• The method successfully learns an infinite set of diverse, task-satisfying behaviors in continuous control environments such as Ant and Half-Cheetah.
• The diversity of behaviors is significantly higher than baseline methods, with clear separation in policy behavior across the latent space.
• The approach achieves data efficiency, requiring fewer environment interactions to converge compared to standard RL algorithms.
• The learned policies generalize well to unseen tasks, enabling fast adaptation through simple latent code sampling.
• Few-shot adaptation performance improves substantially when using diverse solutions, outperforming single-policy baselines.
• The use of continuous latent variables enables smoother and more interpretable behavior interpolation between solutions.