[Paper Review] Meta-Transfer Learning through Hard Tasks

Qianru Sun, Yaoyao Liu+3 more|arXiv (Cornell University)|Oct 7, 2019

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Meta-Transfer Learning (MTL), a novel meta-learning framework that enables deep neural networks (DNNs) to adapt quickly to few-shot tasks by learning task-specific scaling and shifting functions for pre-trained DNN weights. By introducing a hard task (HT) meta-batch curriculum, MTL achieves state-of-the-art performance on miniImageNet, tieredImageNet, and Fewshot-CIFAR100, with up to 1.85% accuracy gain on FC100 and convergence in only 8k tasks—dramatically fewer than MAML’s 240k tasks.

ABSTRACT

Meta-learning has been proposed as a framework to address the challenging few-shot learning setting. The key idea is to leverage a large number of similar few-shot tasks in order to learn how to adapt a base-learner to a new task for which only a few labeled samples are available. As deep neural networks (DNNs) tend to overfit using a few samples only, typical meta-learning models use shallow neural networks, thus limiting its effectiveness. In order to achieve top performance, some recent works tried to use the DNNs pre-trained on large-scale datasets but mostly in straight-forward manners, e.g., (1) taking their weights as a warm start of meta-training, and (2) freezing their convolutional layers as the feature extractor of base-learners. In this paper, we propose a novel approach called meta-transfer learning (MTL) which learns to transfer the weights of a deep NN for few-shot learning tasks. Specifically, meta refers to training multiple tasks, and transfer is achieved by learning scaling and shifting functions of DNN weights for each task. In addition, we introduce the hard task (HT) meta-batch scheme as an effective learning curriculum that further boosts the learning efficiency of MTL. We conduct few-shot learning experiments and report top performance for five-class few-shot recognition tasks on three challenging benchmarks: miniImageNet, tieredImageNet and Fewshot-CIFAR100 (FC100). Extensive comparisons to related works validate that our MTL approach trained with the proposed HT meta-batch scheme achieves top performance. An ablation study also shows that both components contribute to fast convergence and high accuracy.

Motivation & Objective

• Address the challenge of overfitting in few-shot learning when using deep neural networks (DNNs) with limited labeled data.
• Overcome the limitations of existing meta-learning methods that rely on shallow networks or inefficient fine-tuning of DNNs.
• Leverage pre-trained DNNs as a knowledge base to improve few-shot generalization without full fine-tuning.
• Design an efficient training curriculum using hard tasks to accelerate convergence and improve performance.
• Demonstrate the generalization of MTL across different architectures and learning paradigms, including supervised and semi-supervised few-shot learning.

Proposed method

• Propose Meta-Transfer Learning (MTL), which learns task-specific scaling and shifting functions for pre-trained DNN weights instead of fine-tuning all parameters.
• Apply scaling and shifting (SS) operations to both the feature extractor Θ and classifier θ, enabling efficient adaptation with minimal learnable parameters.
• Introduce a hard task (HT) meta-batch scheme that prioritizes challenging tasks during meta-training to act as a curriculum, improving learning efficiency.
• Train the meta-learner using a two-loop optimization: inner loop adapts the base-learner on a few-shot task, outer loop updates meta-parameters using validation performance.
• Use pre-trained ResNet-12, ResNet-18, and ResNet-25 as backbone networks, with only the scaling and shifting parameters updated during meta-training.
• Generalize the MTL framework to both supervised and semi-supervised few-shot learning by adapting the SS operations to different base-learner types.

Experimental results

Research questions

• RQ1 Can meta-transfer learning with scaling and shifting functions on pre-trained DNNs outperform standard fine-tuning and meta-learning baselines in few-shot learning?
• RQ2 Does the proposed hard task (HT) meta-batch curriculum accelerate convergence and improve accuracy compared to conventional meta-batch sampling?
• RQ3 How effective is MTL when applied to different DNN architectures (e.g., ResNet-12, ResNet-18, ResNet-25) in few-shot settings?
• RQ4 Can MTL consistently improve performance across both supervised and semi-supervised few-shot learning paradigms?
• RQ5 To what extent does the HT meta-batch scheme enhance learning efficiency on more challenging benchmarks like FC100?

Key findings

• MTL with SS[Θ;θ] achieves state-of-the-art performance on miniImageNet, tieredImageNet, and Fewshot-CIFAR100, with a 10.6% accuracy gain over No meta-learning on 1-shot miniImageNet.
• The HT meta-batch scheme boosts accuracy by 0.6%–2.2% across benchmarks, with a 1.85% average gain on the more challenging FC100 dataset.
• MTL converges in only ~8k tasks on miniImageNet and ~6k tasks on FC100, a significant improvement over MAML’s 240k tasks.
• Freezing the pre-trained feature extractor Θ and only meta-learning the classifier θ (via SS) outperforms full fine-tuning by ~5% on 1-shot miniImageNet.
• The HT meta-batch consistently improves validation performance across all models and benchmarks, with the largest gains observed on FC100.
• The MTL framework generalizes well: it improves performance across both classical supervised and state-of-the-art semi-supervised few-shot learning models.