[Paper Review] Universal Successor Representations for Transfer Reinforcement Learning

Chen Ma, Junfeng Wen+1 more|arXiv (Cornell University)|Apr 11, 2018

Reinforcement Learning in Robotics8 references18 citations

TL;DR

This paper proposes Universal Successor Representations (USR) and a trainable USR approximator (USRA) to enable efficient transfer learning in reinforcement learning, where tasks share the same dynamics but differ in goals. By learning a shared representation of state transitions and goals, USRA enables fast adaptation to new goals through effective initialization, significantly outperforming random initialization in training speed.

ABSTRACT

The objective of transfer reinforcement learning is to generalize from a set of previous tasks to unseen new tasks. In this work, we focus on the transfer scenario where the dynamics among tasks are the same, but their goals differ. Although general value function (Sutton et al., 2011) has been shown to be useful for knowledge transfer, learning a universal value function can be challenging in practice. To attack this, we propose (1) to use universal successor representations (USR) to represent the transferable knowledge and (2) a USR approximator (USRA) that can be trained by interacting with the environment. Our experiments show that USR can be effectively applied to new tasks, and the agent initialized by the trained USRA can achieve the goal considerably faster than random initialization.

Motivation & Objective

• To address the challenge of transferring knowledge across reinforcement learning tasks with shared dynamics but different goals.
• To improve upon general value function approximators, which are difficult to train effectively in practice.
• To develop a universal successor representation (USR) that generalizes over both states and goals for multi-task transfer.
• To design a trainable USR approximator (USRA) that can be learned via on-policy actor-critic interaction with the environment.
• To demonstrate that USRA enables faster learning on unseen goals through effective initialization.

Proposed method

• Factorize the reward function as $ r_g(s,a,s') = \mathbf{\phi}(s,a,s')^\top \mathbf{w}_g $, where $ \mathbf{\phi} $ are shared state features and $ \mathbf{w}_g $ are goal-specific reward features.
• Define the universal successor representation (USR) as $ \mathbf{\psi}_g^\pi(s) = \mathbb{E}^\pi[\mathbf{\phi}(s,A,S') + \gamma_g(s)\mathbf{\psi}_g^\pi(S')] $, which generalizes over both states and goals.
• Train the USRA using an actor-critic framework with gradient updates on four loss components: $ L_w $, $ L_\psi $, $ J_\pi $, and $ L_{\text{recon}} $ for state feature learning.
• Use a deep neural network architecture where $ \theta_\pi $, $ \theta_\psi $, $ \theta_w $, and $ \theta_\phi $ are jointly optimized, sharing early layers for feature extraction.
• Learn state features $ \mathbf{\phi}(s) $ via autoencoder pre-training on raw observations before end-to-end training.
• Use the trained USRA as an initialization for policy and value function to accelerate learning on new, unseen goals.

Experimental results

Research questions

• RQ1 Can a universal successor representation (USR) effectively generalize across different goals in tasks with shared dynamics?
• RQ2 Can the USR approximator (USRA) be successfully trained through on-policy interaction with the environment?
• RQ3 Does initializing the agent with a pre-trained USRA lead to faster convergence on new, unseen goals compared to random initialization?
• RQ4 How many source goals are required for USRA to achieve strong generalization and fast transfer performance?
• RQ5 Can the USRA-based initialization outperform standard value function transfer methods in multi-task reinforcement learning settings?

Key findings

• The USRA model generalizes effectively across goals, with performance on unseen target goals approaching that of models trained directly on those goals.
• When trained on 20 out of 64 goals, the USRA achieved generalization performance comparable to that trained on 40 goals, indicating low sample complexity for transfer.
• Agents initialized with the trained USRA learned faster on new target goals than those with random initialization, especially when the number of source goals was sufficient (e.g., k=20).
• The model achieved low Mean Squared Error (MSE) between predicted and optimal USR values, and low cross-entropy loss for policy generalization on unseen goals.
• The performance gain from USRA initialization was most pronounced when the number of source goals was large enough to capture task dynamics, with diminishing returns beyond a certain point.
• The actor-critic training procedure successfully optimized all components of the USRA, including $ \theta_\psi $, $ \theta_\pi $, $ \theta_w $, and $ \theta_\phi $, in a unified framework.