[Paper Review] Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation

Haoxiang Wang, Han Zhao+1 more|arXiv (Cornell University)|Jun 16, 2021

Domain Adaptation and Few-Shot Learning63 references19 citations

TL;DR

This paper establishes a theoretical and empirical connection between multi-task learning (MTL) and gradient-based meta-learning (GBML), showing they share the same optimization formulation and produce nearly identical predictions on unseen tasks for over-parameterized networks. The key contribution is a first-order MTL method that achieves state-of-the-art few-shot classification performance while being an order of magnitude faster than second-order GBML methods like MetaOptNet.

ABSTRACT

Multi-task learning (MTL) aims to improve the generalization of several related tasks by learning them jointly. As a comparison, in addition to the joint training scheme, modern meta-learning allows unseen tasks with limited labels during the test phase, in the hope of fast adaptation over them. Despite the subtle difference between MTL and meta-learning in the problem formulation, both learning paradigms share the same insight that the shared structure between existing training tasks could lead to better generalization and adaptation. In this paper, we take one important step further to understand the close connection between these two learning paradigms, through both theoretical analysis and empirical investigation. Theoretically, we first demonstrate that MTL shares the same optimization formulation with a class of gradient-based meta-learning (GBML) algorithms. We then prove that for over-parameterized neural networks with sufficient depth, the learned predictive functions of MTL and GBML are close. In particular, this result implies that the predictions given by these two models are similar over the same unseen task. Empirically, we corroborate our theoretical findings by showing that, with proper implementation, MTL is competitive against state-of-the-art GBML algorithms on a set of few-shot image classification benchmarks. Since existing GBML algorithms often involve costly second-order bi-level optimization, our first-order MTL method is an order of magnitude faster on large-scale datasets such as mini-ImageNet. We believe this work could help bridge the gap between these two learning paradigms, and provide a computationally efficient alternative to GBML that also supports fast task adaptation.

Motivation & Objective

• To understand the theoretical and practical relationship between multi-task learning (MTL) and meta-learning.
• To address the trade-off between training efficiency in MTL and fast adaptation capability in meta-learning.
• To develop a computationally efficient alternative to gradient-based meta-learning (GBML) that supports fast adaptation to unseen tasks.
• To empirically validate that MTL can achieve competitive performance with state-of-the-art GBML algorithms on few-shot benchmarks.

Proposed method

• Theoretical analysis shows that MTL and a class of GBML algorithms share the same bi-level optimization formulation.
• For over-parameterized neural networks with sufficient depth, the predictive functions of MTL and GBML are proven to be close in function space.
• Theoretical gap between MTL and GBML predictions is shown to be inversely proportional to network depth.
• Empirical validation uses synthetic data to demonstrate increasing similarity in predictions as network depth increases.
• A first-order MTL implementation is developed and evaluated on mini-ImageNet and other few-shot benchmarks.
• The method uses joint training with shared representations and fine-tunes a linear classifier on features for meta-testing.

Experimental results

Research questions

• RQ1 Do MTL and GBML share the same underlying optimization formulation?
• RQ2 Can MTL achieve similar predictive performance to GBML on unseen tasks for over-parameterized networks?
• RQ3 Is the prediction gap between MTL and GBML inversely proportional to network depth?
• RQ4 Can a first-order MTL method outperform second-order GBML methods in both accuracy and training speed?
• RQ5 Does MTL support fast adaptation to unseen few-shot tasks while maintaining training efficiency?

Key findings

• MTL and GBML share the same optimization formulation, with GBML being a regularized variant of MTL’s joint training.
• For over-parameterized networks, the predictive functions of MTL and GBML are close, with the gap decreasing as network depth increases.
• On synthetic data, the prediction similarity between MTL and GBML increases with network depth, validating the theoretical analysis.
• On mini-ImageNet, the proposed MTL method achieves comparable or better few-shot classification accuracy than MetaOptNet, a state-of-the-art GBML method.
• The MTL method is an order of magnitude faster to train than MetaOptNet due to first-order optimization, without sacrificing performance.
• The method generalizes to residual ReLU networks and those with LayerNorm, as shown by theoretical extension.