[Paper Review] AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

Ximeng Sun, Rameswar Panda+2 more|arXiv (Cornell University)|Nov 27, 2019

Domain Adaptation and Few-Shot Learning67 references121 citations

TL;DR

AdaShare learns task-specific execution policies to determine which network layers to share or specialize across multiple tasks, achieving competitive accuracy with significantly fewer parameters and lower computation.

ABSTRACT

Multi-task learning is an open and challenging problem in computer vision. The typical way of conducting multi-task learning with deep neural networks is either through handcrafted schemes that share all initial layers and branch out at an adhoc point, or through separate task-specific networks with an additional feature sharing/fusion mechanism. Unlike existing methods, we propose an adaptive sharing approach, called AdaShare, that decides what to share across which tasks to achieve the best recognition accuracy, while taking resource efficiency into account. Specifically, our main idea is to learn the sharing pattern through a task-specific policy that selectively chooses which layers to execute for a given task in the multi-task network. We efficiently optimize the task-specific policy jointly with the network weights, using standard back-propagation. Experiments on several challenging and diverse benchmark datasets with a variable number of tasks well demonstrate the efficacy of our approach over state-of-the-art methods. Project page: https://cs-people.bu.edu/sunxm/AdaShare/project.html.

Motivation & Objective

• Motivate the need for efficient multi-task learning (MTL) that scales with the number of tasks.
• Develop a differentiable method to decide which layers to share across tasks and which to specialize.
• Jointly optimize task performance and resource efficiency without reinforcement learning.
• Provide a training strategy and losses that encourage sharing while maintaining accuracy.

Proposed method

• Model a task-specific binary policy for each network layer to decide whether a layer is executed for a given task.
• Use Gumbel-Softmax sampling to enable differentiable learning of discrete share/skip decisions.
• Train network weights and policy logits jointly via back-propagation with task-specific losses and regularizations.
• Introduce sparsity and sharing losses to encourage compact, share-friendly architectures.
• Apply curriculum-like warm-up and progressive tightening of the policy space to stabilize optimization.
• Maintain a small additional parameter footprint by learning policies directly on layer logits rather than a separate policy network.

Experimental results

Research questions

• RQ1 Can we learn per-task execution paths within a single multi-task network to optimize both accuracy and efficiency?
• RQ2 Do learned task-specific policies effectively share low- to mid-level features while allowing task-specific blocks where needed?
• RQ3 How do sparsity and sharing regularizations affect the compactness and performance of the multi-task model?
• RQ4 What is the comparative performance and parameter efficiency of AdaShare against state-of-the-art MTL approaches across diverse datasets?
• RQ5 How does the approach scale with increasing numbers of tasks and domains?

Key findings

• AdaShare often achieves the best or near-best performance across multiple metrics on NYU v2, CityScapes, Tiny-Taskonomy, DomainNet, and NLP datasets.
• The method reduces parameters by about 50-80% compared to many baselines while maintaining or surpassing accuracy.
• Learned policies frequently favor sharing mid-level blocks (e.g., conv3_x in ResNet) among task groups to maximize positive transfer and limit negative transfer.
• Policy visualizations reveal task correlations align with intuitive domain similarities, guiding sharing patterns.
• AdaShare delivers 7.67%-18.71% FLOP savings on average and up to 80% parameter reductions with competitive or better accuracy.