[Paper Review] TADAM: Task dependent adaptive metric for improved few-shot learning

Boris N. Oreshkin, Pau Rodríguez+1 more|arXiv (Cornell University)|May 23, 2018

Domain Adaptation and Few-Shot Learning34 references199 citations

TL;DR

TADAM introduces metric scaling and a task-conditioned adaptive metric space with auxiliary co-training, achieving state-of-the-art results on mini-Imagenet for few-shot classification.

ABSTRACT

Few-shot learning has become essential for producing models that generalize from few examples. In this work, we identify that metric scaling and metric task conditioning are important to improve the performance of few-shot algorithms. Our analysis reveals that simple metric scaling completely changes the nature of few-shot algorithm parameter updates. Metric scaling provides improvements up to 14% in accuracy for certain metrics on the mini-Imagenet 5-way 5-shot classification task. We further propose a simple and effective way of conditioning a learner on the task sample set, resulting in learning a task-dependent metric space. Moreover, we propose and empirically test a practical end-to-end optimization procedure based on auxiliary task co-training to learn a task-dependent metric space. The resulting few-shot learning model based on the task-dependent scaled metric achieves state of the art on mini-Imagenet. We confirm these results on another few-shot dataset that we introduce in this paper based on CIFAR100. Our code is publicly available at https://github.com/ElementAI/TADAM.

Motivation & Objective

• Investigate how metric scaling affects optimization and performance in few-shot learning.
• Develop a task-conditioned feature extractor to form a task-dependent metric space.
• Enable end-to-end training with auxiliary co-training to improve generalization.
• Demonstrate improvements on challenging few-shot datasets in image classification.

Proposed method

• Introduce a learnable temperature parameter alpha to scale the distance metric in softmax-based classification.
• Show theory on gradient behavior under alpha, identifying two regimes (alpha -> 0 and alpha -> infinity) and their impact on updates.
• Condition the feature extractor on a task representation via a task embedding network that predicts layer-wise gamma and beta (FILM/conditional batch norm style).
• Use the mean class prototype as the task representation to generate task-conditioned parameters.
• Apply auxiliary task co-training to stabilize training and improve generalization, annealing the auxiliary task probability.
• Evaluate with ResNet-12 backbone on mini-Imagenet and a new FC100-based Fewshot-CIFAR100 dataset.

Experimental results

Research questions

• RQ1 Does metric scaling improve few-shot classification performance across different metrics (e.g., Euclidean, cosine)?
• RQ2 Can task conditioning improve the learned metric space for few-shot tasks, and where in the network is it most beneficial?
• RQ3 Does auxiliary co-training help train the task-conditioned model effectively?
• RQ4 What is the impact of the scaling parameter alpha on learning dynamics and final accuracy?

Key findings

• Scaled similarity metrics can match or surpass unscaled metrics, with scaling achieving up to 14 percentage points improvement over baseline in some settings.
• The proposed task-conditioned metric space yields state-of-the-art results on mini-Imagenet 5-way 5-shot (76.7% accuracy) and improves further over prior state-of-the-art by 4.8 percentage points.
• The optimal alpha lies between two asymptotic regimes, and there exists an alpha value that consistently improves performance across datasets; alpha can be cross-validated on a validation set.
• Task conditioning (TEN plus FILM-style parameter generation) alone is not sufficient; auxiliary co-training is crucial to enable effective joint learning of the TEN and the feature extractor.
• On mini-Imagenet, the approach achieves 58.5% (1-shot), 76.7% (5-shot), and 80.8% (10-shot) with scaled Euclidean distance and TEN.
• The method generalizes to FC100-based Fewshot-CIFAR100, showing substantial gains over related prototypical approaches.