[Paper Review] Gradient Surgery for Multi-Task Learning

Tianhe Yu, Saurabh Kumar+4 more|arXiv (Cornell University)|Jan 19, 2020

Domain Adaptation and Few-Shot Learning56 references108 citations

TL;DR

Proposes PCGrad, a gradient-surgery method that projects conflicting task gradients onto the normal plane of other task gradients to mitigate interference in multi-task learning, improving data efficiency and performance across supervised and reinforcement learning tasks.

ABSTRACT

While deep learning and deep reinforcement learning (RL) systems have demonstrated impressive results in domains such as image classification, game playing, and robotic control, data efficiency remains a major challenge. Multi-task learning has emerged as a promising approach for sharing structure across multiple tasks to enable more efficient learning. However, the multi-task setting presents a number of optimization challenges, making it difficult to realize large efficiency gains compared to learning tasks independently. The reasons why multi-task learning is so challenging compared to single-task learning are not fully understood. In this work, we identify a set of three conditions of the multi-task optimization landscape that cause detrimental gradient interference, and develop a simple yet general approach for avoiding such interference between task gradients. We propose a form of gradient surgery that projects a task's gradient onto the normal plane of the gradient of any other task that has a conflicting gradient. On a series of challenging multi-task supervised and multi-task RL problems, this approach leads to substantial gains in efficiency and performance. Further, it is model-agnostic and can be combined with previously-proposed multi-task architectures for enhanced performance.

Motivation & Objective

• Identify optimization challenges in multi-task learning caused by gradient interference.
• Characterize the tragic triad of conflicting gradients, dominating gradients, and high curvature.
• Develop a gradient-surgery technique (PCGrad) to mitigate gradient conflict.
• Demonstrate model-agnostic applicability of PCGrad across supervised and reinforcement learning tasks.

Proposed method

• Define the three conditions of the multi-task optimization landscape that cause gradient interference.
• Introduce PCGrad, which projects a task gradient onto the normal plane of another task gradient when their cosine similarity is negative.
• Provide a simple algorithm (Algorithm 1) to apply PCGrad within any gradient-based optimizer.
• Theoretically analyze PCGrad in two-task convex and differentiable settings and derive sufficient conditions for improved optimization.
• Demonstrate compatibility and improvements when combining PCGrad with existing multi-task architectures (e.g., MTAN, routing networks).
• Evaluate PCGrad on multi-task supervised learning, multi-task RL, and goal-conditioned RL to assess data efficiency and performance.

Experimental results

Research questions

• RQ1 Does PCGrad reduce gradient interference and improve data efficiency across multi-task learning scenarios?
• RQ2 Can PCGrad be combined with existing multi-task architectures to yield further performance gains?
• RQ3 Is the proposed gradient-triads (conflicting gradients, dominating gradients, high curvature) a major factor in optimization difficulty for multi-task learning?
• RQ4 How does PCGrad perform in both supervised and reinforcement learning settings compared to baselines?

Key findings

• PCGrad substantially improves data efficiency and final performance across multi-task supervised learning and multi-task RL problems.
• In CIFAR-100, combining PCGrad with routing networks yields a 2.8 percentage point absolute improvement in test accuracy.
• On CelebA, PCGrad achieves better average multi-task classification error than the prior method Sener and Koltun (8.69 vs. 8.95).
• MTAN with PCGrad achieves best scores in 8 of 9 categories on NYUv2 tasks.
• In Meta-World MT10/MT50 benchmarks, SAC+PCGrad outperforms baselines in success rates and data efficiency, solving more tasks with fewer samples.