[Paper Review] Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD

Fan Chen, Parikshit Ram+1 more|arXiv (Cornell University)|Sep 15, 2021

Domain Adaptation and Few-Shot Learning36 references4 citations

TL;DR

This paper proposes Sign-MAML, a computationally efficient, first-order Model-Agnostic Meta-Learning algorithm that leverages signSGD as the lower-level optimizer in a bilevel optimization framework. By reformulating MAML through signSGD-oriented bilevel optimization, Sign-MAML achieves superior few-shot classification accuracy over FO-MAML with identical computational cost and reduced training time compared to standard MAML.

ABSTRACT

We propose a new computationally-efficient first-order algorithm for Model-Agnostic Meta-Learning (MAML). The key enabling technique is to interpret MAML as a bilevel optimization (BLO) problem and leverage the sign-based SGD(signSGD) as a lower-level optimizer of BLO. We show that MAML, through the lens of signSGD-oriented BLO, naturally yields an alternating optimization scheme that just requires first-order gradients of a learned meta-model. We term the resulting MAML algorithm Sign-MAML. Compared to the conventional first-order MAML (FO-MAML) algorithm, Sign-MAML is theoretically-grounded as it does not impose any assumption on the absence of second-order derivatives during meta training. In practice, we show that Sign-MAML outperforms FO-MAML in various few-shot image classification tasks, and compared to MAML, it achieves a much more graceful tradeoff between classification accuracy and computation efficiency.

Motivation & Objective

• Address the high computational cost of second-order derivatives in standard MAML during meta-training.
• Develop a first-order MAML variant that maintains theoretical grounding without relying on second-order gradient approximations.
• Improve the accuracy-efficiency tradeoff in few-shot image classification compared to FO-MAML and MAML.
• Explore the impact of lower-level optimizer choice on bilevel optimization efficiency and convergence in meta-learning.
• Demonstrate that sign-based optimization can yield a more robust and efficient meta-learning algorithm than standard first-order approximations.

Proposed method

• Reformulate MAML as a bilevel optimization (BLO) problem, where the upper level optimizes meta-parameters and the lower level adapts to task-specific data.
• Replace the standard gradient descent in the lower-level optimization with signSGD, which uses only the sign of gradients to update parameters.
• Theoretical analysis shows that using signSGD in the lower-level unrolls the optimization path into a first-order alternating scheme, eliminating the need for second-order derivatives.
• The resulting algorithm, Sign-MAML, performs only first-order gradient computations during meta-training, matching the computational efficiency of FO-MAML.
• The method naturally avoids meta-gradient estimation errors associated with Hessian approximation in FO-MAML by relying on sign-based descent.
• Sign-MAML maintains model-agnosticity and is applicable to any differentiable model, enabling broad use in vision and NLP tasks.

Experimental results

Research questions

• RQ1 Can signSGD be effectively used as a lower-level optimizer in MAML to achieve first-order, computationally efficient meta-learning?
• RQ2 Does using signSGD in the lower-level optimization lead to a theoretically sound and stable meta-learning algorithm without second-order derivative assumptions?
• RQ3 How does Sign-MAML compare to FO-MAML and MAML in terms of classification accuracy and training efficiency across few-shot learning benchmarks?
• RQ4 Does the performance gain of Sign-MAML over FO-MAML hold across increasingly challenging few-shot settings (e.g., higher N-way, lower K-shot)?
• RQ5 Can Sign-MAML achieve better accuracy than MAML while reducing training time, especially in large-scale or complex few-shot scenarios?

Key findings

• On FS-CIFAR100, Sign-MAML outperforms FO-MAML by 4.8% and 3.7% in 5-way 1-shot and 5-way 5-shot classification, respectively, with similar computation cost.
• For 10-way 1-shot classification on MiniImageNet, Sign-MAML achieves 7.5% higher accuracy than FO-MAML, with the performance gap widening as task difficulty increases.
• Sign-MAML reduces training time by approximately 50% compared to standard MAML while maintaining or improving test accuracy.
• Across varying fine-tuning steps (1 to 5), Sign-MAML consistently outperforms FO-MAML in accuracy, with the improvement remaining stable and not increasing computational cost.
• The accuracy improvement of Sign-MAML over FO-MAML is more pronounced in challenging settings (e.g., high N, low K), with gains up to 8%, while FO-MAML only slightly outperforms Sign-MAML in easier cases by 1–3%.
• Sign-MAML achieves a more graceful tradeoff between accuracy and computation than both MAML and FO-MAML, making it suitable for resource-constrained few-shot learning applications.