[Paper Review] Expressive Priors in Bayesian Neural Networks: Kernel Combinations and Periodic Functions

Tim Pearce, Russell Tsuchida+3 more|arXiv (Cornell University)|May 15, 2019

Gaussian Processes and Bayesian Inference33 references14 citations

TL;DR

This paper introduces a method to design expressive priors in Bayesian neural networks (BNNs) by combining activation functions and architectural components to mimic kernel combinations in Gaussian processes. It demonstrates that multiplying a periodic BNN with a tanh BNN enables precise modeling of locally periodic functions, significantly improving performance in time series and reinforcement learning tasks compared to standard BNNs.

ABSTRACT

A simple, flexible approach to creating expressive priors in Gaussian process (GP) models makes new kernels from a combination of basic kernels, e.g. summing a periodic and linear kernel can capture seasonal variation with a long term trend. Despite a well-studied link between GPs and Bayesian neural networks (BNNs), the BNN analogue of this has not yet been explored. This paper derives BNN architectures mirroring such kernel combinations. Furthermore, it shows how BNNs can produce periodic kernels, which are often useful in this context. These ideas provide a principled approach to designing BNNs that incorporate prior knowledge about a function. We showcase the practical value of these ideas with illustrative experiments in supervised and reinforcement learning settings.

Motivation & Objective

• To address the lack of prior design flexibility in Bayesian neural networks (BNNs), which typically rely on default architectures without incorporating domain-specific knowledge.
• To bridge the gap between Gaussian process (GP) kernel engineering and BNN architecture design by transferring the idea of kernel combinations to BNNs.
• To develop a principled method for constructing BNNs that can express complex functional properties such as periodicity and trends, informed by prior knowledge.
• To demonstrate the practical benefits of these structured priors in real-world learning tasks, including supervised time series prediction and reinforcement learning.

Proposed method

• The paper derives BNN architectures that mirror kernel combinations used in Gaussian processes, such as summing or multiplying basic kernels (e.g., linear, periodic, RBF).
• It introduces a novel BNN component that combines cosine/sin input warping with a ReLU network to model periodic functions, emulating the periodic kernel in GPs.
• A key innovation is the use of a multiplicative architecture: a periodic BNN is multiplied by a separate tanh BNN with a long length scale to model locally periodic behavior.
• The method leverages the theoretical equivalence between infinite-width BNNs and GPs to ensure that the resulting prior functions match desired kernel-induced properties.
• The approach is validated using scalable Bayesian inference via Bayesian ensembles, enabling training on large datasets with hundreds of thousands of experience transitions.
• The architecture is designed to be independent of inference method, making it broadly applicable across different learning frameworks.

Experimental results

Research questions

• RQ1 Can BNN architectures be designed to express the same expressive prior functions as kernel combinations in Gaussian processes?
• RQ2 How can periodic functions be effectively modeled in BNNs, beyond simply using cosine activations?
• RQ3 Does incorporating structured priors—such as periodicity and trend components—improve generalization and sample efficiency in learning tasks?
• RQ4 Can such structured BNN priors outperform standard BNNs in tasks with known functional properties, such as time series or control tasks?

Key findings

• The Periodic × Tanh BNN architecture significantly outperformed both a standard ReLU BNN and a basic Periodic BNN in a pendulum control task, achieving faster learning and higher final cumulative rewards.
• The Periodic × Tanh model captured the true local periodicity of the pendulum dynamics more accurately than other BNNs, as shown in Q-value visualizations across three revolutions.
• The Periodic BNN alone plateaued early in learning, indicating insufficient flexibility to model the full system dynamics, while the ReLU BNN learned slowly but eventually surpassed it.
• The multiplicative architecture enabled precise modeling of locally periodic functions, demonstrating that BNNs can emulate the periodic kernel from GPs through architectural design.
• The results confirm that incorporating prior knowledge via structured BNN design reduces data requirements and improves learning efficiency, consistent with the 'blessing of abstraction' principle.