[Paper Review] Discovering Diverse Nearly Optimal Policies with Successor Features

Tom Zahavy, Brendan O’Donoghue+4 more|arXiv (Cornell University)|Jun 1, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Diverse Successive Policies, a method that discovers a set of near-optimal reinforcement learning policies with high behavioral diversity by optimizing for diversity in successor features (SFs) while maintaining near-optimal performance via a constrained Markov decision process (CMDP). The approach uses explicit diversity rewards that minimize correlation between SFs, outperforming prior methods like robustness and discrimination in discovering qualitatively distinct locomotion behaviors such as three-legged standing or high-kneed walking.

ABSTRACT

Finding different solutions to the same problem is a key aspect of intelligence associated with creativity and adaptation to novel situations. In reinforcement learning, a set of diverse policies can be useful for exploration, transfer, hierarchy, and robustness. We propose Diverse Successive Policies, a method for discovering policies that are diverse in the space of Successor Features, while assuring that they are near optimal. We formalize the problem as a Constrained Markov Decision Process (CMDP) where the goal is to find policies that maximize diversity, characterized by an intrinsic diversity reward, while remaining near-optimal with respect to the extrinsic reward of the MDP. We also analyze how recently proposed robustness and discrimination rewards perform and find that they are sensitive to the initialization of the procedure and may converge to sub-optimal solutions. To alleviate this, we propose new explicit diversity rewards that aim to minimize the correlation between the Successor Features of the policies in the set. We compare the different diversity mechanisms in the DeepMind Control Suite and find that the type of explicit diversity we are proposing is important to discover distinct behavior, like for example different locomotion patterns.

Motivation & Objective

• To address the limitation of most RL methods that focus on finding a single optimal policy, rather than a diverse set of qualitatively different solutions.
• To formalize the discovery of diverse, near-optimal policies as a Constrained Markov Decision Process (CMDP) with diversity in successor features as the primary objective.
• To analyze the sensitivity and sub-optimality of existing intrinsic diversity mechanisms—robustness and discrimination—under different initializations.
• To propose new explicit diversity rewards that minimize correlation between successor features of policies to improve diversity and avoid local minima.
• To demonstrate the effectiveness of the method in learning diverse, high-reward behaviors in the DeepMind Control Suite, including novel locomotion patterns.

Proposed method

• Formalize the problem as a CMDP where the objective is to maximize diversity in successor features (SFs), subject to a constraint that policies remain near-optimal in terms of extrinsic reward.
• Define an intrinsic diversity reward based on minimizing the correlation between the SFs of different policies in the set, using a linear combination of features.
• Use a Lagrangian relaxation approach to jointly optimize for diversity and near-optimality, with adaptive weighting of the constraint.
• Train policies incrementally, adding one policy at a time to the set, each solving a CMDP with a diversity objective derived from the SFs of previously trained policies.
• Apply the method in the DeepMind Control Suite environments, using both sparse-reward and no-reward settings to evaluate diversity and performance.
• Compare the proposed explicit diversity rewards against prior methods (robustness, discrimination) in terms of behavioral diversity and convergence stability.

Experimental results

Research questions

• RQ1 Can a constrained MDP framework with diversity in successor features produce a set of diverse, near-optimal policies?
• RQ2 How do existing intrinsic diversity mechanisms—robustness and discrimination—perform in terms of behavioral diversity and convergence stability?
• RQ3 Does explicitly minimizing correlation between successor features lead to better diversity than implicit or indirect diversity signals?
• RQ4 Can the proposed method discover qualitatively distinct locomotion behaviors (e.g., three-legged standing, high-kneed walking) while maintaining near-optimal performance?
• RQ5 How does the method perform in the no-reward setting, where no extrinsic reward is available during training?

Key findings

• The proposed explicit diversity reward, which minimizes correlation between successor features, significantly outperforms robustness and discrimination methods in discovering diverse locomotion behaviors.
• In the Walker.walk environment, the method discovered distinct walking patterns such as lifting one leg, high-kneed walking, and heel-to-toe walking, achieving 98% of the maximum extrinsic reward.
• In the Dog.stand environment, the method discovered three-legged standing variations, while the baseline with no diversity mechanism produced only four-legged standing, despite similar performance (92–106% of maximum reward).
• In the no-reward setting, the method discovered complex locomotion skills such as backward kneeling, crawling, and flick-flack jumps in Walker, and forward/backward running and jumping in Cheetah.
• The method achieved high diversity with minimal hyperparameter tuning, while robustness and discrimination methods often converged to sub-optimal or static behaviors like 'Yoga' poses.
• All discovered policies satisfied the near-optimality constraint (≥90% of maximum extrinsic reward), confirming that diversity does not come at the cost of performance.