[Paper Review] Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning

Sharu Theresa Jose, Sangwoo Park+1 more|arXiv (Cornell University)|Jun 1, 2021

Gaussian Processes and Bayesian Inference41 references4 citations

TL;DR

This paper presents an information-theoretic analysis of epistemic uncertainty in Bayesian meta-learning, deriving exact characterizations and bounds for the minimum excess meta-risk (MEMR) using conditional mutual information. It shows that epistemic uncertainty scales as $ O(d/log(N)/N) $ from hyperparameter uncertainty and $ O(d/ ext{log}(m)/m) $ from model parameter uncertainty, with experiments confirming meta-learning reduces uncertainty when per-task data is limited but not when data is abundant.

ABSTRACT

The overall predictive uncertainty of a trained predictor can be decomposed into separate contributions due to epistemic and aleatoric uncertainty. Under a Bayesian formulation, assuming a well-specified model, the two contributions can be exactly expressed (for the log-loss) or bounded (for more general losses) in terms of information-theoretic quantities (Xu and Raginsky, 2020). This paper addresses the study of epistemic uncertainty within an information-theoretic framework in the broader setting of Bayesian meta-learning. A general hierarchical Bayesian model is assumed in which hyperparameters determine the per-task priors of the model parameters. Exact characterizations (for the log-loss) and bounds (for more general losses) are derived for the epistemic uncertainty -quantified by the minimum excess meta-risk (MEMR)- of optimal meta-learning rules. This characterization is leveraged to bring insights into the dependence of the epistemic uncertainty on the number of tasks and on the amount of per-task training data. Experiments are presented that use the proposed information-theoretic bounds, evaluated via neural mutual information estimators, to compare the performance of conventional learning and meta-learning as the number of meta-learning tasks increases.

Motivation & Objective

• To extend information-theoretic uncertainty decomposition from conventional Bayesian learning to Bayesian meta-learning.
• To characterize epistemic uncertainty in meta-learning using the minimum excess meta-risk (MEMR) under log-loss.
• To quantify how epistemic uncertainty depends on the number of meta-training tasks $ N $ and per-task data $ m $.
• To evaluate the proposed bounds using neural mutual information estimators (e.g., C-MINE) in a Bayesian neural network setting.
• To compare the performance of conventional learning and meta-learning as $ N $ increases, identifying regimes where meta-learning provides benefit.

Proposed method

• Formalizes Bayesian meta-learning using a hierarchical model with hyperparameter $ U $, per-task parameter $ W $, and task-specific data $ \mathbf{Z} $, assuming a well-specified model.
• Derives an exact information-theoretic characterization of MEMR for log-loss as the sum of two conditional mutual information terms: $ I(Y;W|X,\mathbf{Z},U) $ and $ I(Y;U|X,\mathbf{Z}_{1:N}) $.
• Establishes upper bounds on MEMR for general loss functions using information-theoretic tools, generalizing the minimum excess risk (MER) concept.
• Proposes to estimate the relevant conditional mutual information terms using neural estimators, specifically C-MINE and SMILE, to evaluate bounds empirically.
• Designs experiments with Bayesian neural networks (BNNs) to compare MEMR and its bounds across varying $ N $ and $ m $, using a 3-layer MLP with ReLU activations.
• Employs a 3-layer classifier in C-MINE with ReLU activations and Adam optimization to estimate mutual information between model parameters and training data.

Experimental results

Research questions

• RQ1 How does epistemic uncertainty in Bayesian meta-learning scale with the number of meta-training tasks $ N $ and per-task data $ m $?
• RQ2 What is the information-theoretic structure of the minimum excess meta-risk (MEMR) in hierarchical Bayesian models?
• RQ3 In what regimes does meta-learning significantly reduce epistemic uncertainty compared to conventional learning?
• RQ4 How do the contributions of hyperparameter-level and model parameter-level uncertainty to MEMR evolve with $ N $ and $ m $?
• RQ5 Can neural mutual information estimators accurately evaluate the derived information-theoretic bounds in a BNN setting?

Key findings

• Epistemic uncertainty due to hyperparameter uncertainty scales as $ O(d\log(N)/N) $, decreasing with more meta-training tasks.
• Epistemic uncertainty due to model parameter uncertainty scales as $ O(d\log(m)/m) $, decreasing with more per-task data.
• When per-task data $ m $ is small (e.g., $ m < 10 $), hyperparameter-level sensitivity dominates, and meta-learning with large $ N $ significantly reduces MEMR.
• When per-task data $ m $ is large (e.g., $ m > 75 $), model parameter-level sensitivity dominates, and increasing $ N $ provides marginal benefit over conventional learning.
• The non-vanishing MEMR in the limit of large $ N $ is due to persistent uncertainty from limited per-task data $ m $, which is independent of $ N $.
• Experiments using C-MINE and SMILE estimators confirm that MEMR and its upper bound decrease with $ N $ and $ m $, with the bounds closely tracking the true MEMR in all regimes.