[Paper Review] A Closer Look at Few-shot Classification Again

Xu Luo, Hao Wu+4 more|arXiv (Cornell University)|Jan 28, 2023

Domain Adaptation and Few-Shot Learning16 citations

TL;DR

The paper proves training and adaptation in few-shot classification are disentangled and uncorrelated, enabling independent analysis of each phase, and uncovers neural scaling laws and adaptation dynamics.

ABSTRACT

Few-shot classification consists of a training phase where a model is learned on a relatively large dataset and an adaptation phase where the learned model is adapted to previously-unseen tasks with limited labeled samples. In this paper, we empirically prove that the training algorithm and the adaptation algorithm can be completely disentangled, which allows algorithm analysis and design to be done individually for each phase. Our meta-analysis for each phase reveals several interesting insights that may help better understand key aspects of few-shot classification and connections with other fields such as visual representation learning and transfer learning. We hope the insights and research challenges revealed in this paper can inspire future work in related directions. Code and pre-trained models (in PyTorch) are available at https://github.com/Frankluox/CloserLookAgainFewShot.

Motivation & Objective

• Investigate whether the training and adaptation phases in few-shot classification are completely uncorrelated.
• Provide a systematic analysis of each phase by fixing the other, to understand their separate contributions.
• Characterize how dataset scale, model architecture, and task setup affect performance in both phases.
• Reveal practical insights for algorithm design and connections to representation learning and transfer learning.

Proposed method

• Define a formal framework for evaluating the interaction between training and adaptation algorithms via Avg and CI across algorithm pairs.
• Construct Mtrain and Madapt with diverse learning strategies (meta-learning and non-meta-learning for training; various adaptation methods for adaptation).
• Empirically assess uncorrelatedness by checking if algorithm rankings are preserved across the other phase using Meta-Dataset and multiple backbones.
• Analyze training phase effects by scaling training data (samples per class and number of classes) on ImageNet-scale data with CE, PN, and MoCo-v2.
• Analyze adaptation phase effects by varying ways/shots and backbone adaptation strategies on ImageNet and Quick Draw datasets.

Experimental results

Research questions

• RQ1 Are training and adaptation algorithms uncorrelated in few-shot classification, such that changing one phase does not alter the performance ranking of algorithms in the other phase?
• RQ2 How does the scale of the training dataset (in terms of samples per class and number of classes) affect few-shot learning performance for different training algorithms?
• RQ3 How do ImageNet-level pretrained models correlate with few-shot performance for supervised vs self-supervised training?
• RQ4 What is the impact of different adaptation strategies (finetune, partial finetune, and query-support methods) across varying ways and shots?
• RQ5 What practical guidance emerges for selecting training and adaptation components to maximize few-shot performance?

Key findings

• Training and adaptation algorithms form ordered sets; changing one phase yields monotonic or overlapping improvements across the other phase.
• For supervised models, increasing the number of training classes yields stronger gains than increasing samples per class, indicating a neural scaling law with respect to classes.
• ImageNet performance is not a good predictor of few-shot performance for supervised models, but it strongly predicts performance for self-supervised models.
• Vanilla finetuning often outperforms partial finetuning methods in few-shot adaptation, with advantages growing with more ways/shots and greater task shifts, though at higher time cost.
• Backbone adaptation becomes advantageous for high-way, high-shot, or cross-domain tasks, while query-support matching methods scale poorly as data scale increases.