[Paper Review] Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?

Yonglong Tian, Yue Wang+3 more|arXiv (Cornell University)|Mar 25, 2020

Domain Adaptation and Few-Shot Learning57 references88 citations

TL;DR

A simple baseline that pre-trains a strong embedding on merged meta-training data, freezes it, and trains a linear classifier for each task can outperform many meta-learning methods; self-distillation further boosts performance.

ABSTRACT

The focus of recent meta-learning research has been on the development of learning algorithms that can quickly adapt to test time tasks with limited data and low computational cost. Few-shot learning is widely used as one of the standard benchmarks in meta-learning. In this work, we show that a simple baseline: learning a supervised or self-supervised representation on the meta-training set, followed by training a linear classifier on top of this representation, outperforms state-of-the-art few-shot learning methods. An additional boost can be achieved through the use of self-distillation. This demonstrates that using a good learned embedding model can be more effective than sophisticated meta-learning algorithms. We believe that our findings motivate a rethinking of few-shot image classification benchmarks and the associated role of meta-learning algorithms. Code is available at: http://github.com/WangYueFt/rfs/.

Motivation & Objective

• Motivate a rethinking of few-shot image classification benchmarks and the role of meta-learning algorithms.
• Investigate whether a learned embedding can outperform sophisticated meta-learning methods.
• Propose a simple baseline: pre-train an embedding on merged meta-training data and train a linear classifier on top.
• Show that self-distillation further improves performance.
• Demonstrate that self-supervised embeddings can match supervised ones for few-shot tasks.

Proposed method

• Merge all meta-training tasks into a single large classification task to learn an embedding f_phi via cross-entropy loss on the merged data.
• Freeze the embedding and train a linear base learner (e.g., logistic regression) on top of the fixed embeddings during meta-testing.
• Optionally apply self-distillation (Born-again strategy) to generate multiple generations of embeddings and distill knowledge across generations.
• Evaluate base learners such as logistic regression, nearest neighbors, and cosine-based classifiers on top of frozen embeddings.
• Compare supervised versus self-supervised pre-trained embeddings for downstream few-shot tasks.

Experimental results

Research questions

• RQ1 Can a fixed, well-learned embedding outperform state-of-the-art meta-learning algorithms in few-shot classification?
• RQ2 Does pre-training on a merged meta-training set suffice without fine-tuning on meta-testing data?
• RQ3 Does sequential self-distillation consistently improve few-shot performance across datasets and backbones?

Key findings

• A simple baseline with a fixed embedding often outperforms prior meta-learning methods by large margins.
• Self-distillation provides an additional 2–3% average improvement over the simple baseline on several benchmarks.
• Embeddings learned via state-of-the-art self-supervised methods can achieve similar performance to supervised embeddings for few-shot tasks.
• On Meta-Dataset, the method outperforms previous best results by more than 7% on average.
• Both small (ConvNet) and large (ResNet) backbones benefit from embeddings; better embeddings are key to performance.