[Paper Review] Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer

Lucas N. Alegre, Ana L. C. Bazzan+1 more|arXiv (Cornell University)|Jun 22, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes SFOLS, a novel algorithm that combines successor features (SFs) with Optimistic Linear Support (OLS) to construct a set of policies whose SFs form a convex coverage set (CCS), enabling optimal policy transfer for any new linearly-expressed task via generalized policy improvement (GPI) without additional environment interaction. The method guarantees optimal performance across all linear reward functions by leveraging theoretical equivalence between transfer learning via SFs and multi-objective RL.

ABSTRACT

In many real-world applications, reinforcement learning (RL) agents might have to solve multiple tasks, each one typically modeled via a reward function. If reward functions are expressed linearly, and the agent has previously learned a set of policies for different tasks, successor features (SFs) can be exploited to combine such policies and identify reasonable solutions for new problems. However, the identified solutions are not guaranteed to be optimal. We introduce a novel algorithm that addresses this limitation. It allows RL agents to combine existing policies and directly identify optimal policies for arbitrary new problems, without requiring any further interactions with the environment. We first show (under mild assumptions) that the transfer learning problem tackled by SFs is equivalent to the problem of learning to optimize multiple objectives in RL. We then introduce an SF-based extension of the Optimistic Linear Support algorithm to learn a set of policies whose SFs form a convex coverage set. We prove that policies in this set can be combined via generalized policy improvement to construct optimal behaviors for any new linearly-expressible tasks, without requiring any additional training samples. We empirically show that our method outperforms state-of-the-art competing algorithms both in discrete and continuous domains under value function approximation.

Motivation & Objective

• To address the limitation of existing successor feature (SF)-based methods, which do not guarantee optimal policies for new tasks despite enabling policy transfer.
• To establish a theoretical equivalence between SF-based transfer learning and multi-objective reinforcement learning (MORL) under linear preferences.
• To develop an SF-based extension of the Optimistic Linear Support (OLS) algorithm that constructs a convex coverage set (CCS) of policies with optimal generalization across all linearly-expressed tasks.
• To prove that policies derived from this CCS can be combined via generalized policy improvement (GPI) to yield optimal behaviors for any new task without additional environment interaction.
• To empirically validate SFOLS against state-of-the-art methods in both discrete and continuous domains under value function approximation.

Proposed method

• The method maps the SF-based transfer learning problem to a multi-objective MDP (MOMDP) under linear preferences, establishing theoretical equivalence between the two frameworks.
• It introduces SFOLS, an SF-based extension of OLS that iteratively selects tasks to solve such that the resulting SFs form a convex coverage set (CCS).
• The algorithm uses GPI to combine policies from the CCS, ensuring optimal behavior for any new linear reward function.
• It proves that the resulting policy set is optimal for all linearly-expressible tasks and provides a theoretical bound on performance gap when the CCS is incomplete.
• The method leverages successor features to represent state visitation patterns and enables efficient policy composition without retraining.
• It is shown to solve the unsupervised skill discovery open problem by constructing a CCS over successor representations.

Experimental results

Research questions

• RQ1 Can the problem of optimal policy transfer via successor features be formally mapped to multi-objective reinforcement learning under linear preferences?
• RQ2 Can an SF-based extension of OLS be designed to construct a convex coverage set (CCS) of policies that guarantees optimality for any new linear reward function?
• RQ3 Is it possible to prove that generalized policy improvement (GPI) over such a CCS yields optimal policies without additional environment interaction?
• RQ4 Can SFOLS outperform state-of-the-art methods in both discrete and continuous control domains under value function approximation?
• RQ5 Does SFOLS solve the open problem of optimal skill discovery in the unsupervised setting, as defined by Eysenbach et al. (2022)?

Key findings

• SFOLS constructs a set of policies whose successor features form a convex coverage set (CCS), ensuring that any new linearly-expressible task can be solved optimally via GPI without further environment interaction.
• The method provides theoretical guarantees that the resulting policies are optimal for all linear reward functions, under mild assumptions.
• Empirical results show SFOLS outperforms state-of-the-art algorithms in both discrete and continuous control domains under value function approximation.
• The performance gap between the GPI solution and the optimal policy is bounded when the CCS is incomplete, providing a measure of robustness.
• SFOLS is shown to solve the open problem of optimal skill discovery in the unsupervised setting by constructing a CCS over successor representations.
• The method is the first to formally guarantee optimal policy transfer across all linearly-expressed tasks using SFs and GPI.