[Paper Review] Gradient-Based Meta-Learning Using Uncertainty to Weigh Loss for Few-Shot Learning

Lin Ding, Peng Liu+3 more|arXiv (Cornell University)|Aug 17, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a gradient-based meta-learning method that improves few-shot learning by using homoscedastic uncertainty to dynamically weigh meta-losses, enabling better generalization and robustness. By learning task-specific initialization parameters and adaptively assigning loss weights, the model achieves 1–2% higher accuracy than state-of-the-art methods on miniImageNet and tieredImageNet, with improved resilience to learning rate and query set variations.

ABSTRACT

Model-Agnostic Meta-Learning (MAML) is one of the most successful meta-learning techniques for few-shot learning. It uses gradient descent to learn commonalities between various tasks, enabling the model to learn the meta-initialization of its own parameters to quickly adapt to new tasks using a small amount of labeled training data. A key challenge to few-shot learning is task uncertainty. Although a strong prior can be obtained from meta-learning with a large number of tasks, a precision model of the new task cannot be guaranteed because the volume of the training dataset is normally too small. In this study, first,in the process of choosing initialization parameters, the new method is proposed for task-specific learner adaptively learn to select initialization parameters that minimize the loss of new tasks. Then, we propose two improved methods for the meta-loss part: Method 1 generates weights by comparing meta-loss differences to improve the accuracy when there are few classes, and Method 2 introduces the homoscedastic uncertainty of each task to weigh multiple losses based on the original gradient descent,as a way to enhance the generalization ability to novel classes while ensuring accuracy improvement. Compared with previous gradient-based meta-learning methods, our model achieves better performance in regression tasks and few-shot classification and improves the robustness of the model to the learning rate and query sets in the meta-test set.

Motivation & Objective

• To address task uncertainty in few-shot learning, where small training sets lead to poor generalization despite meta-learning.
• To improve meta-learning robustness by dynamically weighting losses based on task-specific uncertainty.
• To enhance model generalization and performance on few-shot classification and regression tasks.
• To reduce sensitivity to hyperparameters such as learning rate and query set size during meta-testing.
• To develop a method that learns optimal loss weights automatically, avoiding manual tuning.

Proposed method

• Introduces a task-specific learner that selects initialization parameters minimizing loss for new tasks, improving adaptation to task-specific characteristics.
• Proposes a weight generator that computes loss weights based on differences in meta-loss across tasks to prioritize informative tasks.
• Introduces homoscedastic uncertainty as a learnable, task-dependent weighting mechanism for multiple losses in meta-loss optimization.
• Uses gradient descent with uncertainty-based loss weighting to improve generalization during meta-training.
• Employs a meta-loss function where uncertainty estimates act as adaptive weights, balancing contributions from different tasks.
• Trains the model to jointly optimize initialization parameters and uncertainty estimates, enabling dynamic loss balancing.

Experimental results

Research questions

• RQ1 Can dynamic loss weighting based on meta-loss differences improve few-shot classification performance?
• RQ2 Can homoscedastic uncertainty be effectively used to balance losses across tasks in gradient-based meta-learning?
• RQ3 Does uncertainty-based loss weighting improve robustness to variations in learning rate and query set size?
• RQ4 How does the proposed method compare to MAML in terms of accuracy and generalization on few-shot benchmarks?
• RQ5 Can task-specific initialization selection enhance model adaptation to new tasks with limited data?

Key findings

• The proposed method with homoscedastic uncertainty achieves 1–2% higher accuracy than state-of-the-art methods on miniImageNet and tieredImageNet for few-shot classification.
• Performance improvement is most significant with four or more tasks, as uncertainty better captures inter-task noise and balances learning.
• The model is more robust to learning rate changes: at α=10⁻¹, it achieves a lower loss (0.315) than MAML, which diverges under the same conditions.
• The method maintains stable performance across different query set sizes (1, 5, 15), while MAML's accuracy drops significantly as query sets increase.
• The weight generator based on meta-loss differences shows performance gains but falls short of the uncertainty-based method, which achieves superior accuracy.
• The model demonstrates improved generalization and reduced sensitivity to meta-test hyperparameters, enhancing practical usability.