[Paper Review] Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML

Aniruddh Raghu, Maithra Raghu+2 more|arXiv (Cornell University)|Sep 19, 2019

Domain Adaptation and Few-Shot LearningComputer Science35 references96 citations

TL;DR

The paper shows that feature reuse, not rapid learning, largely explains MAML’s success and introduces ANIL and NIL as simplified/test-time variants with competitive performance.

ABSTRACT

An important research direction in machine learning has centered around developing meta-learning algorithms to tackle few-shot learning. An especially successful algorithm has been Model Agnostic Meta-Learning (MAML), a method that consists of two optimization loops, with the outer loop finding a meta-initialization, from which the inner loop can efficiently learn new tasks. Despite MAML's popularity, a fundamental open question remains -- is the effectiveness of MAML due to the meta-initialization being primed for rapid learning (large, efficient changes in the representations) or due to feature reuse, with the meta initialization already containing high quality features? We investigate this question, via ablation studies and analysis of the latent representations, finding that feature reuse is the dominant factor. This leads to the ANIL (Almost No Inner Loop) algorithm, a simplification of MAML where we remove the inner loop for all but the (task-specific) head of a MAML-trained network. ANIL matches MAML's performance on benchmark few-shot image classification and RL and offers computational improvements over MAML. We further study the precise contributions of the head and body of the network, showing that performance on the test tasks is entirely determined by the quality of the learned features, and we can remove even the head of the network (the NIL algorithm). We conclude with a discussion of the rapid learning vs feature reuse question for meta-learning algorithms more broadly.

Motivation & Objective

• Investigate whether MAML’s success comes from rapid task-specific adaptation or from reusable features embedded in the meta-initialization.
• Characterize how different network components (body vs head) contribute to few-shot learning performance.
• Develop simplified alternatives (ANIL, NIL) that retain performance while reducing computation.
• Understand how training regimes influence feature quality and task specificity for meta-learning models.

Proposed method

• Perform layer-freezing experiments to assess the necessity of inner-loop updates on different network portions.
• Use representational similarity analyses (CCA, CKA) to measure changes in latent representations before and after inner-loop adaptation.
• Compare MAML and a simplified ANIL variant where only the head is updated in the inner loop.
• Explore NIL at test time by using body representations with cosine similarity for labeling.
• Evaluate across standard few-shot benchmarks (Omniglot, MiniImageNet) and reinforcement learning tasks.

Experimental results

Research questions

• RQ1 Is MAML’s effectiveness due to rapid inner-loop learning or due to reusable features in the meta-initialization?
• RQ2 How do the body (early layers) and head (final layer) of the network contribute to few-shot learning performance?
• RQ3 Can we simplify MAML without loss of accuracy, and what are the trade-offs in computation?
• RQ4 Does training regime (multitask, multiclass, NIL) affect the quality of learned features and task specificity?

Key findings

• Feature reuse is the dominant factor behind MAML’s efficient learning, as shown by minimal performance loss when freezing body layers.
• Convolutional body representations remain highly similar (CCA/CKA > ~0.9) before and after inner-loop updates, while the head changes substantially.
• ANIL (Almost No Inner Loop) matches MAML’s performance on standard benchmarks while significantly reducing computation.
• NIL (No Inner Loop) at test time using only body features and cosine similarity achieves comparable results to MAML/ANIL on several tasks.
• Training with MAML/ANIL yields superior features for the body compared to multiclass, multitask, or random features regimes.
• The head is important during training to learn good features, but at test time the body’s features can suffice for unseen tasks.