[Paper Review] Multi-task Learning with Gradient Guided Policy Specialization

Wenhao Yu, C. Karen Liu+1 more|arXiv (Cornell University)|Sep 23, 2017

Reinforcement Learning in Robotics18 references3 citations

TL;DR

This paper proposes a multi-task reinforcement learning method that improves data efficiency by first jointly training a shared neural network policy across multiple tasks, then selectively specializing it using a gradient-based metric to identify weights with high inter-task disagreement. The approach outperforms joint training alone, independent training, random specialization, and standard multi-task architectures across diverse robotic control tasks with varying task similarity.

ABSTRACT

We present a method for efficient learning of control policies for multiple related robotic motor skills. Our approach consists of two stages, joint training and specialization training. During the joint training stage, a neural network policy is trained with minimal information to disambiguate the motor skills. This forces the policy to learn a common representation of the different tasks. Then, during the specialization training stage we selectively split the weights of the policy based on a per-weight metric that measures the disagreement among the multiple tasks. By splitting part of the control policy, it can be further trained to specialize to each task. To update the control policy during learning, we use Trust Region Policy Optimization with Generalized Advantage Function (TRPOGAE). We propose a modification to the gradient update stage of TRPO to better accommodate multi-task learning scenarios. We evaluate our approach on three continuous motor skill learning problems in simulation: 1) a locomotion task where three single legged robots with considerable difference in shape and size are trained to hop forward, 2) a manipulation task where three robot manipulators with different sizes and joint types are trained to reach different locations in 3D space, and 3) locomotion of a two-legged robot, whose range of motion of one leg is constrained in different ways. We compare our training method to three baselines. The first baseline uses only joint training for the policy, the second trains independent policies for each task, and the last randomly selects weights to split. We show that our approach learns more efficiently than each of the baseline methods.

Motivation & Objective

• To improve data efficiency in training multiple related robotic motor skills using deep reinforcement learning.
• To address the challenge of balancing parameter sharing and task-specific adaptation in multi-task policy learning.
• To develop an automated method for identifying which network weights should be specialized to improve performance across diverse tasks.
• To evaluate the method on tasks with varying degrees of similarity, including highly dissimilar robots.
• To demonstrate that selective specialization based on gradient disagreement outperforms random or fixed architectural specialization.

Proposed method

• The method uses two-stage training: first, a single shared policy is trained jointly on all tasks using Proximal Policy Optimization (PPO).
• A per-weight metric is computed based on the variance of policy gradients across tasks to measure disagreement; higher variance indicates less shared utility.
• Weights with high disagreement are selectively split to create specialized sub-policies during a second training phase.
• Specialization is applied only to the policy network, not the value function, based on the gradient variance metric.
• The approach allows for flexible specialization amounts and can be applied to any number of related tasks with shared architecture.
• The method is evaluated using PPO with surrogate loss, and specialization is applied at a single point during training.

Experimental results

Research questions

• RQ1 Can selective policy specialization based on gradient disagreement improve multi-task reinforcement learning performance compared to joint training alone?
• RQ2 How does the performance of the proposed method vary with different levels of task similarity (e.g., robots of different sizes)?
• RQ3 Does gradient-guided specialization outperform random or fixed architectural specialization in multi-task learning?
• RQ4 Can the method generalize across diverse robotic control tasks with varying degrees of task relatedness?
• RQ5 What is the impact of joint training duration and specialization ratio on final performance?

Key findings

• The proposed method achieved the highest performance across all tested configurations, consistently outperforming joint training alone, independent training, random specialization, and standard multi-task architectures.
• On the hopper task with 3kg and 15kg torso masses (most dissimilar), the method achieved 2302.74 cumulative return, outperforming independent training and joint training.
• On the biped task with forward and backward walking, the method achieved 2562.53 cumulative return with 800k joint training steps and 50% specialization, significantly outperforming baselines.
• As task similarity increased (e.g., 14kg and 15kg hoppers), joint training with gradient-guided specialization outperformed separate training, achieving 2562.53 vs. 2381.18 in the best baseline.
• The method showed robustness across 9 hyperparameter configurations, with one optimal setting (800k joint steps, 50% specialization) yielding strong results across all examples.
• Preliminary tests showed that specializing the value function network did not yield notable gains, suggesting policy specialization alone is sufficient for performance improvement.