[Paper Review] Which Tasks Should Be Learned Together in Multi-task Learning?

Trevor Standley, Amir Zamir+4 more|arXiv (Cornell University)|May 18, 2019

Domain Adaptation and Few-Shot LearningComputer Science65 references68 citations

TL;DR

The paper analyzes how tasks interact in multi-task learning and introduces a framework to group tasks into networks under a fixed inference budget to maximize overall accuracy, showing that task affinity depends on network size, data, and setup.

ABSTRACT

Many computer vision applications require solving multiple tasks in real-time. A neural network can be trained to solve multiple tasks simultaneously using multi-task learning. This can save computation at inference time as only a single network needs to be evaluated. Unfortunately, this often leads to inferior overall performance as task objectives can compete, which consequently poses the question: which tasks should and should not be learned together in one network when employing multi-task learning? We study task cooperation and competition in several different learning settings and propose a framework for assigning tasks to a few neural networks such that cooperating tasks are computed by the same neural network, while competing tasks are computed by different networks. Our framework offers a time-accuracy trade-off and can produce better accuracy using less inference time than not only a single large multi-task neural network but also many single-task networks.

Motivation & Objective

• Characterize how task relationships affect multi-task learning performance.
• Determine when to group tasks within the same network versus separating them.
• Develop a framework to select task groupings that optimize total loss under inference-time constraints.

Proposed method

• Empirically study task cooperation and interference using the Taskonomy dataset with two 5-task sets.
• Evaluate how network size, dataset size, and task similarity influence multi-task affinity.
• Define and solve an optimization problem to select a set of networks solving subsets of tasks within a budget b.
• Propose approximations (ESA and HOA) to reduce training time for candidate networks.
• Compare against baselines including single-task networks and all-tasks-in-one-network.

Experimental results

Research questions

• RQ1 How do task relationships influence multi-task learning performance across different network capacities and data regimes?
• RQ2 Can we automatically assign tasks to networks to maximize total loss under a fixed inference-time budget?
• RQ3 Do transfer-learning affinities predict multi-task affinities across settings?
• RQ4 What are effective approximations to predict network performance and speed up task-grouping search?

Key findings

• Multi-task networks underperform single-task networks with equivalent total compute when using limited capacity, but can outperform when capacity is redistributed or increased.
• Task affinities are highly setting-dependent; larger networks reveal more beneficial groupings, while smaller networks show weaker or different affinities.
• Surface Normals reliably improve the performance of other tasks when trained together, though the Normals task itself often degrades when co-trained.
• Transfer-learning affinities (Taskonomy) do not strongly predict multi-task affinities; no robust correlation across settings.
• A designed task-grouping framework can outperform traditional baselines (single-task, all-tasks-in-one-network) across budgets; optimal groupings adapt to the budget and network capacity.