[Paper Review] On First-Order Meta-Learning Algorithms

Alex Nichol, Joshua Achiam+1 more|arXiv (Cornell University)|Mar 8, 2018

Domain Adaptation and Few-Shot Learning13 references544 citations

TL;DR

The paper analyzes first-order meta-learning methods for learning a fast-adapt initialize, introduces Reptile, and provides theoretical and empirical support showing these first-order methods can match or outperform second-order MAML variants on few-shot tasks.

ABSTRACT

This paper considers meta-learning problems, where there is a distribution of tasks, and we would like to obtain an agent that performs well (i.e., learns quickly) when presented with a previously unseen task sampled from this distribution. We analyze a family of algorithms for learning a parameter initialization that can be fine-tuned quickly on a new task, using only first-order derivatives for the meta-learning updates. This family includes and generalizes first-order MAML, an approximation to MAML obtained by ignoring second-order derivatives. It also includes Reptile, a new algorithm that we introduce here, which works by repeatedly sampling a task, training on it, and moving the initialization towards the trained weights on that task. We expand on the results from Finn et al. showing that first-order meta-learning algorithms perform well on some well-established benchmarks for few-shot classification, and we provide theoretical analysis aimed at understanding why these algorithms work.

Motivation & Objective

• Motivate meta-learning under a distribution of tasks and seek an initialization that enables rapid adaptation on unseen tasks.
• Analyze and compare first-order meta-learning methods to MAML, highlighting implementation simplicity and performance.
• Introduce a new first-order algorithm (Reptile) and relate it to joint training and fast weights concepts.
• Provide theoretical insight into why first-order updates can generalize within tasks and across minibatches.

Proposed method

• Describe the MAML objective and its first-order variant (FOMAML) by ignoring second-order terms in the meta-gradient.
• Introduce Reptile, a first-order algorithm that updates initialization toward task-specific trained weights, with serial and parallel (batched) variants.
• Show that Reptile’s update can be viewed as moving initialization toward a weighted average of task-trained weights, akin to joint training plus a corrective term.
• Provide a Taylor-series based analysis to compare g_MAML, g_FOMAML, and g_Reptile, highlighting the AvgGrad and AvgGradInner components.
• Discuss a 1D sine wave regression case study to illustrate why first-order methods can recover a useful initialization.
• Empirically evaluate on Omniglot and Mini-ImageNet with both non-transductive and transductive settings, and compare inner-loop gradient combinations.

Experimental results

Research questions

• RQ1 Can first-order meta-learning updates (without second-order derivatives) achieve comparable fast-learning performance to MAML on few-shot tasks?
• RQ2 What is the behavior and theoretical justification for Reptile as a simple first-order alternative to MAML?
• RQ3 How do inner-loop gradient combinations and the overlap between inner-loop minibatches affect meta-learning performance?
• RQ4 Does Reptile effectively converge to initializations near the optimal solution manifolds of multiple tasks?
• RQ5 Under what conditions do first-order methods maximize within-task generalization through gradient interactions?

Key findings

• FOMAML and Reptile perform comparably to MAML on several few-shot benchmarks, with Reptile often matching or slightly edging FOMAML on Mini-ImageNet and Omniglot in reported settings.
• Reptile moves the initialization toward task-trained weights and benefits from using multiple inner-loop gradients, showing improvements when employing more inner-loop steps or gradients (especially with gradient summation).
• A Taylor-series analysis shows leading-order terms: AvgGrad minimizes the joint task loss, while AvgGradInner increases inner-task gradient alignment, aiding within-task generalization.
• Reptile’s update is closely related to joint training on the expected loss but with an additional term that tends to align gradients across minibatches within a task, thereby improving generalization.
• Experiments reveal transduction (batch-normalization-based test-time sharing) boosts performance for all methods; inner-loop parameter choices (shared-tail vs separate-tail, batch size) significantly influence FOMAML more than Reptile.
• The sine-wave regression case study demonstrates that MAML and Reptile can learn initial representations enabling fast post-train adaptation that non-meta-learning joint training cannot achieve.