[Paper Review] Deep Multi-task Representation Learning: A Tensor Factorisation Approach

Yongxin Yang, Timothy M. Hospedales|arXiv (Cornell University)|May 20, 2016

Tensor decomposition and applications105 citations

TL;DR

This paper introduces DMTRL, a deep multi-task learning framework that learns per-layer sharing structures via tensor factorisation (Tucker, TT, and Last Axis Flattening), enabling end-to-end knowledge sharing in deep networks without manually specified sharing patterns.

ABSTRACT

Most contemporary multi-task learning methods assume linear models. This setting is considered shallow in the era of deep learning. In this paper, we present a new deep multi-task representation learning framework that learns cross-task sharing structure at every layer in a deep network. Our approach is based on generalising the matrix factorisation techniques explicitly or implicitly used by many conventional MTL algorithms to tensor factorisation, to realise automatic learning of end-to-end knowledge sharing in deep networks. This is in contrast to existing deep learning approaches that need a user-defined multi-task sharing strategy. Our approach applies to both homogeneous and heterogeneous MTL. Experiments demonstrate the efficacy of our deep multi-task representation learning in terms of both higher accuracy and fewer design choices.

Motivation & Objective

• Motivate end-to-end deep multi-task learning beyond shallow linear models.
• Generalise matrix factorisation-based MTL to tensor factorisation for sharing across layers and outputs in DNNs.
• Enable automatic learning of sharing structure at every layer, for both homogeneous and heterogeneous MTL tasks.
• Provide a scalable framework that reduces user-specified sharing design and supports convolutional and fully connected layers.

Proposed method

• Represent task models as high-order tensors that encode shared and task-specific parameters.
• Apply Tucker decomposition, Tensor Train decomposition, or Last Axis Flattening to factorise weight tensors across layers to realise sharing patterns.
• Synthesize weight tensors during forward passes from learned factorised components, enabling backpropagation without non-differentiable steps.
• Extend sharing mechanisms from fully connected to convolutional layers by treating convolutional kernels as higher-order tensors with shared factors.
• Train using standard backpropagation; initialisation can leverage STL-based decomposed weights with a single max reconstruction error hyperparameter (epsilon=10%).

Experimental results

Research questions

• RQ1 How can deep neural networks automatically learn cross-task sharing at every layer in both homogeneous and heterogeneous MTL settings?
• RQ2 Can tensor factorisation provide a principled, data-driven sharing mechanism that reduces manual architecture search in Deep MTL?
• RQ3 How do Tucker, TT, and Last Axis Flattening share parameters across tasks and outputs within DNNs?
• RQ4 Do DMTRL methods outperform single-task learning and match or exceed carefully designed user-defined MTL baselines across diverse datasets?
• RQ5 What is the practical impact of learned sharing on performance and architectural design in deep MTL?

Key findings

• DMTRL methods consistently outperform single-task learning on evaluated tasks.
• DMTRL-Tucker and DMTRL-TT match or exceed the best user-defined MTL architectures when training data are limited, and are comparable when data are abundant.
• The approach learns layer-wise sharing that gradually decreases in higher layers, aligning with intuition that lower layers are more shared than top layers.
• Across homogeneous, heterogeneous, and multilingual letter recognition tasks, DMTRL variants demonstrate robust performance gains over STL and competitive results against UD-MTL.
• The sharing strength can be quantified via the S-like factor matrices, showing deeper layers tend to exhibit less sharing while earlier layers share more.