[Paper Review] A Theoretical Framework for Bayesian Nonparametric Regression: Orthonormal Random Series and Rates of Contraction

Fangzheng Xie, Wei Jin+1 more|arXiv (Cornell University)|Dec 15, 2017

Gaussian Processes and Bayesian Inference42 references3 citations

TL;DR

This paper introduces a Bayesian nonparametric regression framework using orthonormal random series to derive rates of contraction under the integrated $L_2$-distance without requiring uniform boundedness of the function space. It establishes a general theorem that yields near-parametric rates for the squared-exponential Gaussian process on analytic functions, adaptive rates for sieve priors, and adaptive-and-exact rates for the un-modified block prior on H"older-smooth functions.

ABSTRACT

We develop a unifying framework for Bayesian nonparametric regression to study the rates of contraction with respect to the integrated $L_2$-distance without assuming the regression function space to be uniformly bounded. The framework is built upon orthonormal random series in a flexible manner. A general theorem for deriving rates of contraction for Bayesian nonparametric regression is provided under the proposed framework. As specific applications, we obtain the near-parametric rate of contraction for the squared-exponential Gaussian process when the true function is analytic, the adaptive rates of contraction for the sieve prior, and the adaptive-and-exact rates of contraction for the un-modified block prior when the true function is {\alpha}-smooth. Extensions to wavelet series priors and fixed-design regression problems are also discussed.

Motivation & Objective

• To develop a unifying theoretical framework for Bayesian nonparametric regression that avoids the restrictive assumption of uniformly bounded function spaces.
• To establish general conditions under which posterior contraction rates can be derived with respect to the integrated $L_2$-distance.
• To apply the framework to specific priors—such as the Gaussian process, sieve prior, and block prior—to derive sharp contraction rates.
• To extend the framework to wavelet series priors and fixed-design regression settings.
• To achieve adaptive and exact contraction rates for smooth functions under minimal regularity assumptions.

Proposed method

• The framework is built on orthonormal random series expansions, allowing flexible representation of the regression function.
• A general theorem is derived to bound the posterior concentration around the true function in terms of the $L_2$-distance.
• The method avoids uniform boundedness by leveraging the structure of orthonormal bases and tail probability bounds on series coefficients.
• Contraction rates are analyzed via metric entropy and testing arguments, adapted to the orthonormal series setting.
• The framework is applied to specific priors by characterizing the corresponding coefficient priors and their entropy properties.
• Extensions to wavelet series and fixed-design models are achieved by adapting the orthonormal basis and design-specific metric entropy bounds.

Experimental results

Research questions

• RQ1 What are the conditions under which posterior contraction rates can be derived for Bayesian nonparametric regression without assuming bounded function spaces?
• RQ2 How do orthonormal random series representations enable the derivation of sharp contraction rates in the absence of uniform boundedness?
• RQ3 What contraction rates are achieved for the squared-exponential Gaussian process when the true function is analytic?
• RQ4 Can the framework yield adaptive rates for sieve priors and exact rates for block priors under smoothness assumptions?
• RQ5 How can the framework be extended to wavelet series priors and fixed-design regression problems?

Key findings

• The framework achieves the near-parametric rate of contraction for the squared-exponential Gaussian process when the true function is analytic, under the integrated $L_2$-distance.
• Adaptive rates of contraction are established for the sieve prior, matching the optimal rate up to logarithmic factors.
• The un-modified block prior achieves both adaptive and exact contraction rates when the true function is $\alpha$-smooth, without tuning.
• The method extends to wavelet series priors, enabling similar contraction rate analysis in that setting.
• The framework applies to fixed-design regression, providing a general approach for contraction rate analysis in that context.
• The general theorem provides a unified tool for deriving contraction rates across diverse nonparametric priors without requiring boundedness assumptions.