[Paper Review] Sparse Uncertainty Representation in Deep Learning with Inducing Weights

Hippolyt Ritter, Martin Kukla+2 more|arXiv (Cornell University)|May 30, 2021

Gaussian Processes and Bayesian Inference71 references4 citations

TL;DR

This paper proposes inducing weights—low-dimensional auxiliary parameters that project uncertainty quantification into a compressed space—enabling memory-efficient Bayesian neural networks and deep ensembles. By extending Matheron’s conditional Gaussian sampling rule, the method achieves competitive uncertainty estimation and accuracy with up to 75.7% parameter reduction, reducing parameter count to ≤24.3% of a standard network while maintaining strong performance on ResNets and fully connected networks.

ABSTRACT

Bayesian neural networks and deep ensembles represent two modern paradigms of uncertainty quantification in deep learning. Yet these approaches struggle to scale mainly due to memory inefficiency issues, since they require parameter storage several times higher than their deterministic counterparts. To address this, we augment the weight matrix of each layer with a small number of inducing weights, thereby projecting the uncertainty quantification into such low dimensional spaces. We further extend Matheron's conditional Gaussian sampling rule to enable fast weight sampling, which enables our inference method to maintain reasonable run-time as compared with ensembles. Importantly, our approach achieves competitive performance to the state-of-the-art in prediction and uncertainty estimation tasks with fully connected neural networks and ResNets, while reducing the parameter size to $\leq 24.3\%$ of that of a $single$ neural network.

Motivation & Objective

• To address the high memory cost of Bayesian neural networks (BNNs) and deep ensembles, which require significantly more parameters than deterministic networks.
• To extend sparse Gaussian process techniques—known for memory efficiency—to finite-width deep neural networks via inducing weights.
• To enable efficient posterior sampling and uncertainty quantification without sacrificing predictive performance or calibration.
• To provide a parameter-efficient alternative to full BNNs and deep ensembles, especially for deployment on resource-constrained devices.
• To demonstrate that uncertainty can be effectively represented in a low-dimensional subspace without loss of predictive or uncertainty estimation quality.

Proposed method

• Introduces inducing weights as low-dimensional auxiliary variables that project uncertainty from full weight matrices, reducing parameter count.
• Applies variational inference with a fully factorized Gaussian (FFG) posterior over inducing weights instead of full weights.
• Extends Matheron’s rule for conditional Gaussian sampling to enable fast, efficient weight sampling from the posterior distribution.
• Uses a parameterization that projects uncertainty into a lower-dimensional space via a small inducing weight matrix per layer.
• Applies the framework to both Bayesian neural networks and deep ensembles, enabling uncertainty quantification with reduced memory footprint.
• Employs a PyTorch wrapper called 'bayesianize' to convert deterministic models into Bayesian ones with minimal code changes.

Experimental results

Research questions

• RQ1 Can sparse uncertainty representations in deep learning achieve comparable performance to full Bayesian neural networks while drastically reducing parameter count?
• RQ2 Can inducing weights enable efficient and scalable posterior sampling comparable to deep ensembles?
• RQ3 To what extent can uncertainty be accurately represented in a low-dimensional subspace of the weight space?
• RQ4 How does the proposed method compare to standard BNNs and deep ensembles in terms of calibration, accuracy, and robustness under distribution shift?
• RQ5 Can the method be effectively applied to modern architectures like ResNets and achieve parameter efficiency without performance degradation?

Key findings

• The proposed inducing weight method reduces parameter count to ≤24.3% of a standard deterministic network while maintaining competitive performance on image classification tasks.
• On CIFAR-100, the method achieves 11.28% test error with 100% inducing weight usage, comparable to full BNNs and deep ensembles.
• For out-of-distribution detection, the method achieves AUC scores of 90.4% on CIFAR-100 and 91.2% on SVHN, matching or exceeding baseline methods.
• On corrupted CIFAR-100, the inducing weight approach maintains high accuracy (e.g., 58.11% at 25% weight usage) and low ECE (4.64% at 100% usage), showing robustness.
• The method enables fast inference via extended Matheron’s rule, maintaining reasonable runtime compared to standard ensembles.
• The approach generalizes to both BNNs and deep ensembles, demonstrating broad applicability across uncertainty quantification paradigms.