[Paper Review] Walsh-Hadamard Variational Inference for Bayesian Deep Learning

Simone Rossi, Sébastien Marmin+1 more|arXiv (Cornell University)|May 27, 2019

Gaussian Processes and Bayesian Inference63 references4 citations

TL;DR

This paper proposes Walsh-Hadamard Variational Inference (whvi), a method that leverages Walsh-Hadamard transforms to factorize weight matrices in Bayesian deep neural networks, reducing parameterization from O(D²) to O(D) and accelerating computation from O(D²) to O(D log D). The approach mitigates over-regularization in variational inference by inducing a matrix-variate posterior approximation with significantly reduced complexity, achieving state-of-the-art performance on multiple benchmark datasets with substantial speedups.

ABSTRACT

Over-parameterized models, such as DeepNets and ConvNets, form a class of models that are routinely adopted in a wide variety of applications, and for which Bayesian inference is desirable but extremely challenging. Variational inference offers the tools to tackle this challenge in a scalable way and with some degree of flexibility on the approximation, but for over-parameterized models this is challenging due to the over-regularization property of the variational objective. Inspired by the literature on kernel methods, and in particular on structured approximations of distributions of random matrices, this paper proposes Walsh-Hadamard Variational Inference (WHVI), which uses Walsh-Hadamard-based factorization strategies to reduce the parameterization and accelerate computations, thus avoiding over-regularization issues with the variational objective. Extensive theoretical and empirical analyses demonstrate that WHVI yields considerable speedups and model reductions compared to other techniques to carry out approximate inference for over-parameterized models, and ultimately show how advances in kernel methods can be translated into advances in approximate Bayesian inference.

Motivation & Objective

• To address the over-regularization problem in variational inference for over-parameterized models like Bayesian DNNs and CNNs, where the KL divergence term dominates the objective due to high-dimensional parameter spaces.
• To reduce the computational and parameter complexity of variational posterior approximations in deep learning while maintaining or improving model flexibility.
• To leverage structured matrix factorizations inspired by kernel methods—specifically Walsh-Hadamard transforms—to enable scalable and efficient Bayesian inference in deep models.
• To demonstrate that matrix-variate posterior approximations with log-linear complexity can outperform standard mean-field and other factorized approaches in both speed and performance.

Proposed method

• The method uses a structured factorization of weight matrices via the Walsh-Hadamard transform, reducing the number of variational parameters from O(D²) to O(D) for a D×D weight matrix.
• It introduces a reparameterization trick based on the inverse Walsh-Hadamard transform to enable differentiable sampling from the variational posterior.
• A local reparameterization trick is derived to reduce gradient variance, enabling efficient stochastic optimization with O(D log D) computational complexity per layer.
• The posterior is modeled as a matrix-variate Gaussian distribution with a low-rank structure induced by the Walsh-Hadamard transformation, enhancing flexibility without increasing complexity.
• The approach draws on random matrix theory and prior work in kernel methods such as Fastfood and Orthogonal Random Features, adapting them to variational inference in deep learning.
• The method is applied to Bayesian DNNs and CNNs, with experimental validation across multiple classification benchmarks.

Experimental results

Research questions

• RQ1 Can structured matrix factorizations based on Walsh-Hadamard transforms reduce the parameterization and computational cost of variational inference in over-parameterized deep models?
• RQ2 Does the proposed whvi method mitigate the over-regularization issue caused by the KL divergence term in the variational objective?
• RQ3 How does the performance of whvi compare to mean-field variational inference and other matrix factorization techniques in terms of accuracy, uncertainty calibration, and training speed?
• RQ4 Can a matrix-variate posterior approximation with log-linear complexity achieve better performance than standard mean-field approximations in Bayesian deep learning?

Key findings

• On six classification datasets, whvi achieved state-of-the-art test error rates, with error rates as low as 0.01±0.00 on the drive dataset using 256 hidden units.
• For the eeg dataset, whvi achieved a test error of 0.17±0.01 with 512 hidden units, outperforming all baselines including Monte Carlo Dropout and standard mean-field VI.
• On the magic dataset, whvi achieved a test error of 0.13±0.01 across all model sizes, matching or surpassing other methods in both accuracy and uncertainty calibration (MNLL).
• The method reduced the number of parameters from O(D²) to O(D), and computational complexity from O(D²) to O(D log D), enabling significant speedups in training and inference.
• In terms of negative log-likelihood (NLL), whvi achieved the lowest values on multiple datasets, such as 0.31±0.01 on the miniboo dataset with 512 units, indicating superior uncertainty estimation.
• The ablation study confirmed that whvi outperformed other low-rank factorization strategies with the same complexity, demonstrating the effectiveness of the Walsh-Hadamard structure.