[Paper Review] How to train your MAML

Antreas Antoniou, Harrison Edwards+1 more|arXiv (Cornell University)|Oct 22, 2018

Domain Adaptation and Few-Shot Learning27 references83 citations

TL;DR

The paper introduces MAML++, an enhanced variant of MAML that stabilizes training, reduces computation, and improves generalization, achieving state-of-the-art results on standard few-shot benchmarks (Omniglot and Mini-ImageNet).

ABSTRACT

The field of few-shot learning has recently seen substantial advancements. Most of these advancements came from casting few-shot learning as a meta-learning problem. Model Agnostic Meta Learning or MAML is currently one of the best approaches for few-shot learning via meta-learning. MAML is simple, elegant and very powerful, however, it has a variety of issues, such as being very sensitive to neural network architectures, often leading to instability during training, requiring arduous hyperparameter searches to stabilize training and achieve high generalization and being very computationally expensive at both training and inference times. In this paper, we propose various modifications to MAML that not only stabilize the system, but also substantially improve the generalization performance, convergence speed and computational overhead of MAML, which we call MAML++.

Motivation & Objective

• Motivate the need for robust few-shot learning and rapid adaptation with minimal data.
• Identify stability, generalization, and computational bottlenecks in MAML.
• Propose architectural and algorithmic improvements to address these issues.
• Demonstrate improved performance and efficiency on standard few-shot benchmarks.

Proposed method

• Extend MAML with a set of six targeted improvements to stabilize training and boost performance.
• Introduce multi-step loss optimization (MSL) to provide gradients at every inner-loop step.
• Apply derivative-order annealing to switch from first- to second-order gradients during training.
• Use per-step batch normalization running statistics to replace shared statistics across inner steps.
• Adopt per-step batch normalization weights and biases to match changing feature distributions.
• Implement per-layer, per-step learning rates and gradient directions to increase adaptation flexibility while controlling overhead.
• Employ cosine annealing for the meta-optimizer learning rate to avoid hyperparameter searches.

Experimental results

Research questions

• RQ1 Can the instability and high computational overhead of MAML be mitigated without sacrificing generalization?
• RQ2 Do a series of targeted architectural and optimization changes yield better stability, faster convergence, and improved few-shot accuracy?
• RQ3 Is it possible to achieve state-of-the-art results on Omniglot and Mini-ImageNet with a revised MAML framework?
• RQ4 How do per-step statistics, learned per-layer learning rates, and learning-rate scheduling affect meta-learning performance across tasks?

Key findings

• MAML++ stabilizes training across seeds and architectures, with faster convergence and higher generalization compared to original MAML.
• The proposed multi-step loss and annealing strategies improve gradient propagation and final performance.
• Per-step batch normalization statistics and per-step biases improve convergence speed and generalization.
• Learning per-layer per-step learning rates and gradient directions provides flexibility with manageable overhead.
• Cosine-annealed meta-optimizer learning rate reduces need for extensive hyperparameter search while sustaining performance.
• Empirical results on Omniglot and Mini-Imagenet show MAML++ achieving state-of-the-art performance, outperforming prior methods under several settings.