[Paper Review] Bayesian Meta-Learning for the Few-Shot Setting via Deep Kernels

Massimiliano Patacchiola, Jack Turner+3 more|arXiv (Cornell University)|Oct 11, 2019

Domain Adaptation and Few-Shot Learning48 references62 citations

TL;DR

Introduces Deep Kernel Transfer (DKT), a Bayesian meta-learning approach using deep kernels to perform few-shot regression, classification, and cross-domain tasks without inner-loop optimization, while providing uncertainty estimates.

ABSTRACT

Recently, different machine learning methods have been introduced to tackle the challenging few-shot learning scenario that is, learning from a small labeled dataset related to a specific task. Common approaches have taken the form of meta-learning: learning to learn on the new problem given the old. Following the recognition that meta-learning is implementing learning in a multi-level model, we present a Bayesian treatment for the meta-learning inner loop through the use of deep kernels. As a result we can learn a kernel that transfers to new tasks; we call this Deep Kernel Transfer (DKT). This approach has many advantages: is straightforward to implement as a single optimizer, provides uncertainty quantification, and does not require estimation of task-specific parameters. We empirically demonstrate that DKT outperforms several state-of-the-art algorithms in few-shot classification, and is the state of the art for cross-domain adaptation and regression. We conclude that complex meta-learning routines can be replaced by a simpler Bayesian model without loss of accuracy.

Motivation & Objective

• Motivate and address the challenge of learning from very small labeled datasets across related tasks.
• Provide a Bayesian alternative to multi-level meta-learning by marginalizing task-specific parameters.
• Develop a Deep Kernel Transfer (DKT) method that transfers a common kernel across tasks and optimizes hyperparameters via marginal likelihood (ML-II).
• Demonstrate that a simpler hierarchical Bayesian model with deep kernels can match or exceed complex meta-learning methods across regression, cross-domain, and classification benchmarks.

Proposed method

• Model few-shot learning as a hierarchical Bayesian problem with shared cross-task kernel hyperparameters and task-specific latent representations.
• Use deep kernel learning where inputs are mapped through a neural network F_phi to a latent space, then a kernel k'(h,h') provides the covariance for a Gaussian process.
• Replace the inner loop of meta-learning with an analytic Gaussian-process marginal likelihood, optimizing only the cross-task parameters (kernel hyperparameters theta and network weights phi).
• Derive closed-form predictive distributions for regression (Gaussian likelihood) and a regression-based treatment for classification (label regression / least-squares classification).
• Train via stochastic ML-II by maximizing the log marginal likelihood across tasks, sampling tasks per minibatch.

Experimental results

Research questions

• RQ1 Can a single, transferable deep kernel learned across many tasks provide competitive or superior performance in few-shot regression, cross-domain adaptation, and classification?
• RQ2 Does marginalizing task-specific inner-loop parameters via a Bayesian integral (ML-II) yield robust uncertainty estimates and simpler optimization compared to standard meta-learning?
• RQ3 How does Deep Kernel Transfer perform across in-domain, out-of-domain, and cross-domain few-shot scenarios compared to state-of-the-art meta-learning baselines?
• RQ4 What is the impact of different kernels (e.g., RBF vs Spectral) and feature normalization on few-shot performance and uncertainty?

Key findings

• DKT achieves competitive or superior performance compared to several state-of-the-art few-shot methods in regression tasks, including improved handling of out-of-range inputs and uncertainty estimation.
• Using a spectral kernel within DKT yields particularly strong performance on unknown periodic function regression and head pose trajectory tasks.
• DKT demonstrates strong results in cross-domain classification, outperforming many baselines and achieving state-of-the-art among comparable methods on several benchmarks.
• The approach provides uncertainty quantification in predictions, which is demonstrated to be meaningful in low-data regimes and under input corruption.
• A simpler Bayesian model with kernel transfer can match or exceed complex meta-learning routines without sacrificing accuracy.