[Paper Review] Meta-learning the Learning Trends Shared Across Tasks

Jathushan Rajasegaran, Salman Khan+3 more|arXiv (Cornell University)|Oct 19, 2020

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes PAMELA, a model-agnostic meta-learning framework that learns dynamic, task-specific learning trends by optimizing iteration-specific gradient preconditioning and using gradient-skip connections to aggregate historical context. It achieves faster convergence and superior few-shot classification and regression performance by modeling how update directions and learning rates evolve over time across tasks.

ABSTRACT

Meta-learning stands for 'learning to learn' such that generalization to new tasks is achieved. Among these methods, Gradient-based meta-learning algorithms are a specific sub-class that excel at quick adaptation to new tasks with limited data. This demonstrates their ability to acquire transferable knowledge, a capability that is central to human learning. However, the existing meta-learning approaches only depend on the current task information during the adaptation, and do not share the meta-knowledge of how a similar task has been adapted before. To address this gap, we propose a 'Path-aware' model-agnostic meta-learning approach. Specifically, our approach not only learns a good initialization for adaptation, it also learns an optimal way to adapt these parameters to a set of task-specific parameters, with learnable update directions, learning rates and, most importantly, the way updates evolve over different time-steps. Compared to the existing meta-learning methods, our approach offers: (a) The ability to learn gradient-preconditioning at different time-steps of the inner-loop, thereby modeling the dynamic learning behavior shared across tasks, and (b) The capability of aggregating the learning context through the provision of direct gradient-skip connections from the old time-steps, thus avoiding overfitting and improving generalization. In essence, our approach not only learns a transferable initialization, but also models the optimal update directions, learning rates, and task-specific learning trends. Specifically, in terms of learning trends, our approach determines the way update directions shape up as the task-specific learning progresses and how the previous update history helps in the current update. Our approach is simple to implement and demonstrates faster convergence. We report significant performance improvements on a number of FSL datasets.

Motivation & Objective

• Address the limitation of existing meta-learning methods that ignore learning trends (e.g., evolving update directions and step-sizes) during inner-loop adaptation.
• Improve generalization and convergence speed in few-shot learning by encoding shared temporal dynamics of task-specific optimization paths.
• Overcome gradient vanishing and overfitting in inner-loop updates by incorporating historical gradient information through residual-like connections.
• Develop a model-agnostic meta-learning approach that learns not only a good initialization but also optimal adaptation trajectories across tasks.
• Enable faster and more stable adaptation by learning how learning behavior changes over time, mimicking human-like learning progression.

Proposed method

• Introduce iteration-specific gradient preconditioning matrices in the inner-loop, allowing distinct learning dynamics at each update step.
• Design a residual connection-based mechanism that enables direct flow of gradients from previous time-steps, forming gradient-skip connections.
• Learn a shared meta-learner that jointly optimizes the initialization and the update rule, including step-size and direction, across tasks.
• Use a two-loop training paradigm: outer loop updates meta-parameters based on task performance, inner loop applies task-specific adaptation with dynamic preconditioning.
• Integrate the gradient-skip mechanism to preserve context from earlier updates, improving stability and reducing overfitting during adaptation.
• Maintain model-agnosticism by ensuring the method can be applied to any differentiable model without architectural changes.

Experimental results

Research questions

• RQ1 Can learning trends—such as evolving update directions and step-sizes—be effectively modeled across tasks to improve meta-learning performance?
• RQ2 How does incorporating historical gradient information via skip connections affect convergence and generalization in few-shot learning?
• RQ3 Does learning iteration-specific preconditioning improve adaptation speed and accuracy compared to single-preconditioning or fixed update rules?
• RQ4 To what extent do gradient-skip connections mitigate gradient vanishing and overfitting during inner-loop optimization?
• RQ5 How does the proposed method compare to MAML and Meta-SGD in terms of convergence speed, parameter efficiency, and performance on few-shot classification and regression tasks?

Key findings

• PAMELA achieves significant performance gains on few-shot classification benchmarks, including miniImageNet, tieredImageNet, and CIFAR-FS, outperforming MAML and Meta-SGD.
• The model converges faster than MAML and Meta-SGD, with only a 5% increase in training and inference time per task compared to MAML.
• Inner-loop gradients in PAMELA decrease in magnitude over time, indicating stable and efficient adaptation, whereas MAML gradients remain large and divergent.
• The meta-gradients of PAMELA remain near zero, suggesting the model has learned a near-optimal initialization, while MAML requires large meta-updates.
• Gradient-skip connections improve generalization and reduce overfitting, with ablation studies showing their contribution is substantial despite minimal impact on width.
• The method is computationally efficient, with only 33% more parameters than MAML, compared to 100% more in Meta-SGD, making it scalable and practical.