[Paper Review] Learning Shared Representations in Multi-task Reinforcement Learning

Diana Borsa, Thore Graepel+1 more|arXiv (Cornell University)|Mar 7, 2016

Reinforcement Learning in Robotics17 references13 citations

TL;DR

This paper proposes a multi-task reinforcement learning framework that jointly learns shared representations for value functions across multiple tasks using modified Fitted Q-Iteration and Fitted Policy Iteration. By leveraging multi-task supervised learning techniques, the method achieves data efficiency and generalization, learning compact, transferable state-action abstractions that enable fast policy inference even with limited samples.

ABSTRACT

We investigate a paradigm in multi-task reinforcement learning (MT-RL) in which an agent is placed in an environment and needs to learn to perform a series of tasks, within this space. Since the environment does not change, there is potentially a lot of common ground amongst tasks and learning to solve them individually seems extremely wasteful. In this paper, we explicitly model and learn this shared structure as it arises in the state-action value space. We will show how one can jointly learn optimal value-functions by modifying the popular Value-Iteration and Policy-Iteration procedures to accommodate this shared representation assumption and leverage the power of multi-task supervised learning. Finally, we demonstrate that the proposed model and training procedures, are able to infer good value functions, even under low samples regimes. In addition to data efficiency, we will show in our analysis, that learning abstractions of the state space jointly across tasks leads to more robust, transferable representations with the potential for better generalization. this shared representation assumption and leverage the power of multi-task supervised learning. Finally, we demonstrate that the proposed model and training procedures, are able to infer good value functions, even under low samples regimes. In addition to data efficiency, we will show in our analysis, that learning abstractions of the state space jointly across tasks leads to more robust, transferable representations with the potential for better generalization.

Motivation & Objective

• To address the inefficiency of learning tasks in isolation by exploiting shared structure across tasks in multi-task reinforcement learning.
• To develop a method that jointly learns optimal value functions across multiple tasks using shared representations.
• To improve data efficiency and generalization by learning transferable, low-dimensional abstractions of the state-action space.
• To demonstrate that shared representations can implicitly capture option-like behavior without explicit subgoal specification.

Proposed method

• Extends Fitted Q-Iteration (FQI) and Fitted Policy Iteration (FPI) to jointly learn value functions across multiple tasks using a shared representation.
• Models the value function as a linear combination of shared features and task-specific components, enabling joint optimization.
• Leverages multi-task learning algorithms from supervised settings to regularize and improve generalization of the shared representation.
• Uses a shared low-dimensional feature space $\psi_s$ to represent states, which is refined during training to capture transferable structure.
• Employs a shared representation $\psi_s$ that is updated iteratively to reflect improving policies and value functions.
• Applies off-policy learning and experience replay to decouple data collection from value function learning, enhancing sample efficiency.

Experimental results

Research questions

• RQ1 Can shared representations across multiple tasks improve data efficiency in reinforcement learning?
• RQ2 How can standard value-iteration and policy-iteration procedures be adapted to jointly learn value functions under a shared representation?
• RQ3 To what extent can the shared representation capture transferable, option-like behavior without explicit subgoal definitions?
• RQ4 Does joint learning lead to more robust and generalizable policies compared to single-task learning?
• RQ5 Can the learned representation generalize to new, unseen tasks within the same environment?

Key findings

• The proposed method achieves good policy performance with significantly fewer samples per task—demonstrating data efficiency even under low-sample regimes.
• The shared representation enables the inference of option-like policies (e.g., navigating to specific rooms) using only 200 samples per task, with no explicit subgoal specification.
• The method successfully learns value functions that generalize across tasks, with results showing that 10–30 samples suffice for simpler navigation tasks.
• The shared feature space captures a compressed, transferable abstraction of the environment, resembling hierarchical options without requiring predefined subgoals.
• Empirical results show that the joint learning procedure leads to more robust and refined representations compared to single-task baselines.
• The method generalizes to new tasks not seen during training, indicating strong transferability of the shared representation.