[Paper Review] Global Convergence and Induced Kernels of Gradient-Based Meta-Learning with Neural Nets.

Haoxiang Wang, Ruoyu Sun+1 more|arXiv (Cornell University)|Jun 25, 2020

Domain Adaptation and Few-Shot Learning44 references13 citations

TL;DR

This paper establishes global convergence for gradient-based meta-learning (GBML) with over-parameterized deep neural networks (DNNs), proving linear-rate convergence to global optima. It further shows GBML is functionally equivalent to a gradient descent that transfers experience from past tasks, leading to a novel kernel-based meta-learning method that outperforms standard DNNs on Omniglot with limited meta-training tasks.

ABSTRACT

Gradient-based meta-learning (GBML) with deep neural nets (DNNs) has become a popular approach for few-shot learning. However, due to the non-convexity of DNNs and the complex bi-level optimization in GBML, the theoretical properties of GBML with DNNs remain largely unknown. In this paper, we first develop a novel theoretical analysis to answer the following questions: Does GBML with DNNs have global convergence guarantees? We provide a positive answer to this question by proving that GBML with over-parameterized DNNs is guaranteed to converge to global optima at a linear rate. The second question we aim to address is: How does GBML achieve fast adaption to new tasks with experience on past similar tasks? To answer it, we prove that GBML is equivalent to a functional gradient descent operation that explicitly propagates experience from the past tasks to new ones. Finally, inspired by our theoretical analysis, we develop a new kernel-based meta-learning approach. We show that the proposed approach outperforms GBML with standard DNNs on the Omniglot dataset when the number of past tasks for meta-training is small. The code is available at this https URL AI-secure/Meta-Neural-Kernel .

Motivation & Objective

• To establish global convergence guarantees for gradient-based meta-learning (GBML) using over-parameterized deep neural networks (DNNs).
• To explain how GBML enables fast adaptation to new tasks by analyzing the functional mechanism of experience transfer from past tasks.
• To develop a new kernel-based meta-learning approach inspired by theoretical insights, aiming to improve performance when meta-training data is scarce.

Proposed method

• Theoretical analysis of the bi-level optimization in GBML using over-parameterized DNNs to prove global convergence at a linear rate.
• Derivation of an equivalent functional gradient descent formulation that explicitly models experience propagation from past tasks to new ones.
• Design of a new kernel-based meta-learning method by leveraging the functional equivalence identified in the theoretical analysis.
• Implementation of the proposed kernel-based method and evaluation on the Omniglot dataset under low-shot meta-training regimes.
• Use of over-parameterization to ensure the convergence properties hold under realistic DNN training dynamics.
• Comparison of the kernel-based method against standard DNN-based GBML to validate performance gains with limited past tasks.

Experimental results

Research questions

• RQ1 Does gradient-based meta-learning with over-parameterized deep neural networks converge globally to a global optimum, and at what rate?
• RQ2 How does GBML enable fast adaptation to new tasks by leveraging experience from past tasks?
• RQ3 Can the theoretical understanding of GBML as a functional gradient descent operation be leveraged to design a more effective kernel-based meta-learning method?

Key findings

• GBML with over-parameterized DNNs is guaranteed to converge to global optima at a linear rate, resolving a key open question on convergence in non-convex meta-learning settings.
• GBML is functionally equivalent to a gradient descent operation that explicitly propagates knowledge from past tasks, explaining its fast adaptation capability.
• The proposed kernel-based meta-learning method outperforms standard DNN-based GBML on the Omniglot dataset when the number of past tasks for meta-training is small.
• The theoretical analysis reveals that the meta-learner's update rule effectively performs functional gradient descent, providing a new interpretation of GBML's learning dynamics.
• The kernel-based method achieves better few-shot generalization than DNN-based GBML under low-data meta-training conditions, validating the practical utility of the theoretical insights.