[Paper Review] Hierarchical Gaussian Process Priors for Bayesian Neural Network Weights

Theofanis Karaletsos, Thang D. Bui|arXiv (Cornell University)|Feb 10, 2020

Gaussian Processes and Bayesian Inference33 references4 citations

TL;DR

This paper introduces a hierarchical Gaussian process prior over Bayesian neural network weights using unit-level latent variables to model correlated weight structures, enabling flexible inductive biases and input-dependent priors via kernel functions. The method achieves superior uncertainty quantification on out-of-distribution data and competitive performance in active learning, outperforming strong baselines including DKL and MAP estimation.

ABSTRACT

Probabilistic neural networks are typically modeled with independent weight priors, which do not capture weight correlations in the prior and do not provide a parsimonious interface to express properties in function space. A desirable class of priors would represent weights compactly, capture correlations between weights, facilitate calibrated reasoning about uncertainty, and allow inclusion of prior knowledge about the function space such as periodicity or dependence on contexts such as inputs. To this end, this paper introduces two innovations: (i) a Gaussian process-based hierarchical model for network weights based on unit embeddings that can flexibly encode correlated weight structures, and (ii) input-dependent versions of these weight priors that can provide convenient ways to regularize the function space through the use of kernels defined on contextual inputs. We show these models provide desirable test-time uncertainty estimates on out-of-distribution data, demonstrate cases of modeling inductive biases for neural networks with kernels which help both interpolation and extrapolation from training data, and demonstrate competitive predictive performance on an active learning benchmark.

Motivation & Objective

• To address the limitations of independent weight priors in Bayesian neural networks, which fail to capture weight correlations and lack a compact interface for expressing functional inductive biases.
• To develop a structured, parameter-efficient prior over network weights that enables calibrated uncertainty estimation and incorporation of prior knowledge such as periodicity or input context.
• To enable input-dependent weight priors through kernel-based constructions that regularize function space behavior while preserving weight structure.
• To design an efficient variational inference scheme that maintains weight correlations without direct inference in high-dimensional weight space.
• To demonstrate improved performance in interpolation, extrapolation, uncertainty estimation, and active learning compared to strong baselines.

Proposed method

• Proposes a hierarchical prior over neural network weights using unit-level latent variables $\mathbf{z}_{l,i}$, where each weight $w_{l,i,j}$ is generated via a mapping function $f([\mathbf{z}_{l,i}, \mathbf{z}_{l+1,j}])$ from concatenated latent codes.
• Employs a meta-network (hyper-network) $\mathrm{NN}_\theta$ to map latent codes to weight means, with a Gaussian noise model $p(\mathbf{w}|\mathbf{C}_w(\mathbf{z}), \theta) = \mathcal{N}(\mathbf{w}|\mathrm{NN}_\theta(\mathbf{C}_w(\mathbf{z})))$.
• Introduces a global GP prior over unit latent variables $\mathbf{z}$, with $p(\mathbf{z}) = \prod \mathcal{N}(\mathbf{z}_{l,i}; \mathbf{0}, \mathbf{I})$, enabling structured correlation modeling across weights.
• Extends the model to input-dependent priors by defining a local kernel $k_{\text{local}}$ on contextual inputs, allowing per-datapoint priors that encode functional inductive biases.
• Uses structured variational inference to approximate the posterior over latent variables $\mathbf{z}$, avoiding direct inference in high-dimensional weight space while preserving weight correlations.
• Employs a product kernel construction combining global and local kernels to flexibly model input-dependent weight priors and functional properties.

Experimental results

Research questions

• RQ1 Can a hierarchical GP prior over unit-level latent variables effectively model correlated weight structures in Bayesian neural networks?
• RQ2 Can input-dependent weight priors, constructed via kernels on contextual inputs, improve generalization and uncertainty calibration in function space?
• RQ3 Does the proposed structured variational inference strategy maintain weight correlations and outperform mean-field approximations in predictive performance and uncertainty estimation?
• RQ4 Can the model achieve better active learning performance by leveraging informative, structured priors and uncertainty-aware selection criteria?
• RQ5 How does the model compare to strong baselines like DKL and MAP in terms of uncertainty quantification on out-of-distribution data?

Key findings

• The proposed MetaGP model achieves significantly better uncertainty estimates on out-of-distribution data than all comparators, including DKL and MAP, which often produce overconfident predictions.
• On the Kuzushiji-MNIST dataset, the model demonstrates superior uncertainty calibration, with predictive entropy clearly distinguishing OOD samples from in-distribution ones.
• In active learning on UCI regression datasets, the MetaGP model achieves comparable or better predictive performance than mean-field VI with fewer queries, indicating effective uncertainty-driven sampling.
• The model outperforms MAP estimation, which consistently produces poor uncertainty estimates on OOD data, even when it performs well on in-distribution test sets.
• The structured variational inference approach successfully preserves weight correlations and enables efficient training without direct posterior inference over high-dimensional weight spaces.
• The use of product kernels for input-dependent priors allows the model to encode functional inductive biases such as periodicity or context dependence, improving both interpolation and extrapolation.