[Paper Review] DiGrad: Multi-Task Reinforcement Learning with Shared Actions

Parijat Dewangan, S. Phani Teja+3 more|arXiv (Cornell University)|Feb 27, 2018

Reinforcement Learning in Robotics17 references5 citations

TL;DR

This paper proposes DiGrad, a deep reinforcement learning framework for multi-task learning in robotic systems with shared actions in continuous control. By leveraging differential policy gradients and a single shared actor-critic network, DiGrad stabilizes training and improves performance over DDPG, especially in complex manipulators like 8-link planar arms and 27-DoF humanoids, achieving faster convergence and better robustness across tasks with shared kinematic chains.

ABSTRACT

Most reinforcement learning algorithms are inefficient for learning multiple tasks in complex robotic systems, where different tasks share a set of actions. In such environments a compound policy may be learnt with shared neural network parameters, which performs multiple tasks concurrently. However such compound policy may get biased towards a task or the gradients from different tasks negate each other, making the learning unstable and sometimes less data efficient. In this paper, we propose a new approach for simultaneous training of multiple tasks sharing a set of common actions in continuous action spaces, which we call as DiGrad (Differential Policy Gradient). The proposed framework is based on differential policy gradients and can accommodate multi-task learning in a single actor-critic network. We also propose a simple heuristic in the differential policy gradient update to further improve the learning. The proposed architecture was tested on 8 link planar manipulator and 27 degrees of freedom(DoF) Humanoid for learning multi-goal reachability tasks for 3 and 2 end effectors respectively. We show that our approach supports efficient multi-task learning in complex robotic systems, outperforming related methods in continuous action spaces.

Motivation & Objective

• To address instability and performance degradation in multi-task deep reinforcement learning when tasks share common actions in continuous control environments.
• To develop a unified framework that enables efficient, stable, and robust training of multiple tasks using a single compound policy with shared parameters.
• To improve generalization and reduce negative gradient interference in multi-task robotic systems, particularly in branched manipulators with overlapping action spaces.
• To demonstrate the effectiveness of DiGrad on complex robotic systems such as 8-link planar manipulators and 27-DoF humanoids for multi-goal reachability tasks.

Proposed method

• The framework uses a single shared actor-critic network to learn multiple deterministic policies simultaneously, with each task having its own action-value function.
• It introduces a differential policy gradient update rule that computes gradients based only on the action and reward of the corresponding task, minimizing interference between tasks.
• A heuristic normalization of the gradient for shared actions is applied to stabilize training, specifically normalizing the gradient of shared action values as per Equation (11).
• The method supports both single-critic and multi-critic architectures, with empirical results showing the single-critic variant performs better due to improved parameter sharing and correlation learning.
• The framework is built upon the DDPG algorithm but extends it with task-specific value function updates and differential gradient computation to handle multi-task dynamics.
• Experiments are conducted using a shared action space in robotic systems, with reward shaping designed to reflect task-specific progress while avoiding dominance of one task over others.

Experimental results

Research questions

• RQ1 Can a unified actor-critic network with differential policy gradients stabilize multi-task reinforcement learning in continuous control with shared actions?
• RQ2 How does DiGrad compare to DDPG in terms of convergence speed, final performance, and training stability across multiple tasks with overlapping action spaces?
• RQ3 Does normalizing the gradient of shared actions improve training stability and performance in multi-task deep reinforcement learning?
• RQ4 Can a single-critic architecture outperform multi-critic setups in multi-task learning when shared parameters are used?
• RQ5 To what extent does DiGrad prevent negative gradient interference and task dominance in multi-goal robotic control?

Key findings

• DiGrad outperformed DDPG in all tested environments, achieving faster convergence and higher final average rewards on both the 8-link planar manipulator and 27-DoF humanoid robot.
• The single-critic DiGrad framework achieved better performance than multi-critic variants, with fewer parameters and improved correlation learning across tasks.
• The application of the gradient normalization heuristic improved training stability, particularly in early training phases, without significantly altering final performance.
• DDPG exhibited instability and task dominance in multi-task settings, especially when using a summed reward signal, due to negative gradient interference.
• On the 8-link manipulator, DiGrad maintained consistent performance across all three tasks despite differing action dimensions, while DDPG only matched performance on Task-2.
• On the humanoid robot, DiGrad with the heuristic in the single-critic setting achieved the best results in terms of stability and final performance, with no degradation over time.